2023-03-26 13:53:28,891 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18 2023-03-26 13:53:28,906 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-03-26 13:53:28,924 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-26 13:53:28,925 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/cluster_18e80c40-7651-1a2f-6095-ce10559d351d, deleteOnExit=true 2023-03-26 13:53:28,925 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-26 13:53:28,926 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/test.cache.data in system properties and HBase conf 2023-03-26 13:53:28,926 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/hadoop.tmp.dir in system properties and HBase conf 2023-03-26 13:53:28,927 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/hadoop.log.dir in system properties and HBase conf 2023-03-26 13:53:28,927 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-26 13:53:28,927 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-26 13:53:28,928 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-26 13:53:29,045 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-26 13:53:29,462 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-26 13:53:29,467 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-26 13:53:29,468 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-26 13:53:29,468 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-26 13:53:29,469 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-26 13:53:29,469 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-26 13:53:29,470 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-26 13:53:29,470 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-26 13:53:29,471 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-26 13:53:29,471 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-26 13:53:29,472 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/nfs.dump.dir in system properties and HBase conf 2023-03-26 13:53:29,472 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/java.io.tmpdir in system properties and HBase conf 2023-03-26 13:53:29,472 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-26 13:53:29,473 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-26 13:53:29,473 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-26 13:53:30,068 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-26 13:53:30,073 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-26 13:53:30,420 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-26 13:53:30,621 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-26 13:53:30,636 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-26 13:53:30,683 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-26 13:53:30,724 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/java.io.tmpdir/Jetty_localhost_35807_hdfs____.vr6kl5/webapp 2023-03-26 13:53:30,865 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:35807 2023-03-26 13:53:30,904 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-26 13:53:30,905 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-26 13:53:31,402 WARN [Listener at localhost/40005] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-26 13:53:31,497 WARN [Listener at localhost/40005] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-26 13:53:31,516 WARN [Listener at localhost/40005] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-26 13:53:31,523 INFO [Listener at localhost/40005] log.Slf4jLog(67): jetty-6.1.26 2023-03-26 13:53:31,528 INFO [Listener at localhost/40005] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/java.io.tmpdir/Jetty_localhost_33565_datanode____fnt4qj/webapp 2023-03-26 13:53:31,628 INFO [Listener at localhost/40005] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33565 2023-03-26 13:53:31,987 WARN [Listener at localhost/41531] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-26 13:53:32,541 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x7481f427769316df: Processing first storage report for DS-ce026d9f-bfd8-44bb-9c87-03a1980b8fd8 from datanode 8c3178b5-de1c-466e-9d5b-1432272d7f2d 2023-03-26 13:53:32,542 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x7481f427769316df: from storage DS-ce026d9f-bfd8-44bb-9c87-03a1980b8fd8 node DatanodeRegistration(127.0.0.1:42615, datanodeUuid=8c3178b5-de1c-466e-9d5b-1432272d7f2d, infoPort=35013, infoSecurePort=0, ipcPort=41531, storageInfo=lv=-57;cid=testClusterID;nsid=117472725;c=1679838810147), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-03-26 13:53:32,543 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x7481f427769316df: Processing first storage report for DS-b3a4bd2a-580c-48de-b2f1-a3a179b55c8b from datanode 8c3178b5-de1c-466e-9d5b-1432272d7f2d 2023-03-26 13:53:32,543 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x7481f427769316df: from storage DS-b3a4bd2a-580c-48de-b2f1-a3a179b55c8b node DatanodeRegistration(127.0.0.1:42615, datanodeUuid=8c3178b5-de1c-466e-9d5b-1432272d7f2d, infoPort=35013, infoSecurePort=0, ipcPort=41531, storageInfo=lv=-57;cid=testClusterID;nsid=117472725;c=1679838810147), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-26 13:53:32,618 DEBUG [Listener at localhost/41531] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18 2023-03-26 13:53:32,681 INFO [Listener at localhost/41531] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/cluster_18e80c40-7651-1a2f-6095-ce10559d351d/zookeeper_0, clientPort=54332, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/cluster_18e80c40-7651-1a2f-6095-ce10559d351d/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/cluster_18e80c40-7651-1a2f-6095-ce10559d351d/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-26 13:53:32,695 INFO [Listener at localhost/41531] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=54332 2023-03-26 13:53:32,702 INFO [Listener at localhost/41531] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-26 13:53:32,705 INFO [Listener at localhost/41531] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-26 13:53:33,360 INFO [Listener at localhost/41531] util.FSUtils(471): Created version file at hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366 with version=8 2023-03-26 13:53:33,360 INFO [Listener at localhost/41531] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/hbase-staging 2023-03-26 13:53:33,683 INFO [Listener at localhost/41531] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-26 13:53:34,165 INFO [Listener at localhost/41531] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-03-26 13:53:34,197 INFO [Listener at localhost/41531] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-26 13:53:34,197 INFO [Listener at localhost/41531] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-26 13:53:34,197 INFO [Listener at localhost/41531] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-26 13:53:34,197 INFO [Listener at localhost/41531] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-26 13:53:34,198 INFO [Listener at localhost/41531] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-26 13:53:34,344 INFO [Listener at localhost/41531] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-26 13:53:34,416 DEBUG [Listener at localhost/41531] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-26 13:53:34,511 INFO [Listener at localhost/41531] ipc.NettyRpcServer(120): Bind to /172.31.14.131:45485 2023-03-26 13:53:34,522 INFO [Listener at localhost/41531] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-26 13:53:34,524 INFO [Listener at localhost/41531] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-26 13:53:34,545 INFO [Listener at localhost/41531] zookeeper.RecoverableZooKeeper(93): Process identifier=master:45485 connecting to ZooKeeper ensemble=127.0.0.1:54332 2023-03-26 13:53:34,588 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:454850x0, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-26 13:53:34,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:45485-0x101211a03bf0000 connected 2023-03-26 13:53:34,625 DEBUG [Listener at localhost/41531] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-26 13:53:34,625 DEBUG [Listener at localhost/41531] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-26 13:53:34,629 DEBUG [Listener at localhost/41531] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-26 13:53:34,637 DEBUG [Listener at localhost/41531] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45485 2023-03-26 13:53:34,637 DEBUG [Listener at localhost/41531] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45485 2023-03-26 13:53:34,640 DEBUG [Listener at localhost/41531] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45485 2023-03-26 13:53:34,640 DEBUG [Listener at localhost/41531] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45485 2023-03-26 13:53:34,641 DEBUG [Listener at localhost/41531] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45485 2023-03-26 13:53:34,647 INFO [Listener at localhost/41531] master.HMaster(444): hbase.rootdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366, hbase.cluster.distributed=false 2023-03-26 13:53:34,722 INFO [Listener at localhost/41531] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-03-26 13:53:34,722 INFO [Listener at localhost/41531] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-26 13:53:34,723 INFO [Listener at localhost/41531] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-26 13:53:34,723 INFO [Listener at localhost/41531] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-26 13:53:34,723 INFO [Listener at localhost/41531] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-26 13:53:34,723 INFO [Listener at localhost/41531] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-26 13:53:34,728 INFO [Listener at localhost/41531] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-26 13:53:34,731 INFO [Listener at localhost/41531] ipc.NettyRpcServer(120): Bind to /172.31.14.131:42517 2023-03-26 13:53:34,734 INFO [Listener at localhost/41531] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-26 13:53:34,740 DEBUG [Listener at localhost/41531] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-03-26 13:53:34,741 INFO [Listener at localhost/41531] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-26 13:53:34,744 INFO [Listener at localhost/41531] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-26 13:53:34,745 INFO [Listener at localhost/41531] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:42517 connecting to ZooKeeper ensemble=127.0.0.1:54332 2023-03-26 13:53:34,749 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:425170x0, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-26 13:53:34,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:42517-0x101211a03bf0001 connected 2023-03-26 13:53:34,751 DEBUG [Listener at localhost/41531] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-26 13:53:34,752 DEBUG [Listener at localhost/41531] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-26 13:53:34,753 DEBUG [Listener at localhost/41531] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-26 13:53:34,755 DEBUG [Listener at localhost/41531] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=42517 2023-03-26 13:53:34,756 DEBUG [Listener at localhost/41531] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=42517 2023-03-26 13:53:34,756 DEBUG [Listener at localhost/41531] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=42517 2023-03-26 13:53:34,757 DEBUG [Listener at localhost/41531] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=42517 2023-03-26 13:53:34,757 DEBUG [Listener at localhost/41531] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=42517 2023-03-26 13:53:34,759 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,45485,1679838813504 2023-03-26 13:53:34,770 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-26 13:53:34,771 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,45485,1679838813504 2023-03-26 13:53:34,792 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-26 13:53:34,792 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-26 13:53:34,792 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 13:53:34,793 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-26 13:53:34,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-26 13:53:34,796 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,45485,1679838813504 from backup master directory 2023-03-26 13:53:34,800 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,45485,1679838813504 2023-03-26 13:53:34,800 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-26 13:53:34,801 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-26 13:53:34,801 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,45485,1679838813504 2023-03-26 13:53:34,804 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-26 13:53:34,806 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-26 13:53:34,899 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/hbase.id with ID: e5819dc1-2150-45e8-8c2a-361c3c39ee64 2023-03-26 13:53:34,942 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-26 13:53:34,956 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 13:53:34,995 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x46da7ed8 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:53:35,031 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2f71ed98, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:53:35,056 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 13:53:35,058 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-26 13:53:35,079 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-26 13:53:35,079 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-26 13:53:35,082 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:53:35,087 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:53:35,089 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-26 13:53:35,125 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/MasterData/data/master/store-tmp 2023-03-26 13:53:35,153 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:53:35,153 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-26 13:53:35,153 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-26 13:53:35,153 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-26 13:53:35,153 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-26 13:53:35,154 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-26 13:53:35,154 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-26 13:53:35,154 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-26 13:53:35,155 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/MasterData/WALs/jenkins-hbase4.apache.org,45485,1679838813504 2023-03-26 13:53:35,180 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C45485%2C1679838813504, suffix=, logDir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/MasterData/WALs/jenkins-hbase4.apache.org,45485,1679838813504, archiveDir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/MasterData/oldWALs, maxLogs=10 2023-03-26 13:53:35,250 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42615,DS-ce026d9f-bfd8-44bb-9c87-03a1980b8fd8,DISK] 2023-03-26 13:53:35,262 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:53:35,338 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/MasterData/WALs/jenkins-hbase4.apache.org,45485,1679838813504/jenkins-hbase4.apache.org%2C45485%2C1679838813504.1679838815195 2023-03-26 13:53:35,338 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42615,DS-ce026d9f-bfd8-44bb-9c87-03a1980b8fd8,DISK]] 2023-03-26 13:53:35,339 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-26 13:53:35,339 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:53:35,343 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-26 13:53:35,344 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-26 13:53:35,420 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-26 13:53:35,427 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-26 13:53:35,451 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-26 13:53:35,465 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:53:35,471 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-26 13:53:35,473 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-26 13:53:35,489 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-26 13:53:35,499 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-26 13:53:35,500 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61731933, jitterRate=-0.08012251555919647}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-26 13:53:35,500 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-26 13:53:35,502 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-26 13:53:35,521 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-26 13:53:35,521 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-26 13:53:35,523 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-26 13:53:35,525 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-26 13:53:35,557 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 32 msec 2023-03-26 13:53:35,557 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-26 13:53:35,584 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-26 13:53:35,590 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-26 13:53:35,617 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-26 13:53:35,621 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-26 13:53:35,623 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-26 13:53:35,628 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-26 13:53:35,632 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-26 13:53:35,635 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 13:53:35,637 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-26 13:53:35,637 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-26 13:53:35,654 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-26 13:53:35,658 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-26 13:53:35,658 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-26 13:53:35,659 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 13:53:35,659 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,45485,1679838813504, sessionid=0x101211a03bf0000, setting cluster-up flag (Was=false) 2023-03-26 13:53:35,675 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 13:53:35,681 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-26 13:53:35,682 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,45485,1679838813504 2023-03-26 13:53:35,690 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 13:53:35,698 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-26 13:53:35,699 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,45485,1679838813504 2023-03-26 13:53:35,703 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.hbase-snapshot/.tmp 2023-03-26 13:53:35,764 INFO [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(952): ClusterId : e5819dc1-2150-45e8-8c2a-361c3c39ee64 2023-03-26 13:53:35,771 DEBUG [RS:0;jenkins-hbase4:42517] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-26 13:53:35,781 DEBUG [RS:0;jenkins-hbase4:42517] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-26 13:53:35,781 DEBUG [RS:0;jenkins-hbase4:42517] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-26 13:53:35,788 DEBUG [RS:0;jenkins-hbase4:42517] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-26 13:53:35,789 DEBUG [RS:0;jenkins-hbase4:42517] zookeeper.ReadOnlyZKClient(139): Connect 0x0cbf7cd6 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:53:35,799 DEBUG [RS:0;jenkins-hbase4:42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2789331, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:53:35,800 DEBUG [RS:0;jenkins-hbase4:42517] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64717a6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-26 13:53:35,828 DEBUG [RS:0;jenkins-hbase4:42517] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:42517 2023-03-26 13:53:35,833 INFO [RS:0;jenkins-hbase4:42517] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-26 13:53:35,833 INFO [RS:0;jenkins-hbase4:42517] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-26 13:53:35,833 DEBUG [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1023): About to register with Master. 2023-03-26 13:53:35,836 INFO [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,45485,1679838813504 with isa=jenkins-hbase4.apache.org/172.31.14.131:42517, startcode=1679838814721 2023-03-26 13:53:35,852 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-26 13:53:35,857 DEBUG [RS:0;jenkins-hbase4:42517] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-26 13:53:35,862 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-26 13:53:35,862 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-26 13:53:35,862 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-26 13:53:35,862 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-26 13:53:35,863 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-03-26 13:53:35,863 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 13:53:35,863 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-26 13:53:35,863 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 13:53:35,865 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1679838845865 2023-03-26 13:53:35,868 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-26 13:53:35,872 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-26 13:53:35,872 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-26 13:53:35,878 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-26 13:53:35,878 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-26 13:53:35,886 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-26 13:53:35,887 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-26 13:53:35,887 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-26 13:53:35,887 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-26 13:53:35,888 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-26 13:53:35,893 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-26 13:53:35,894 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-26 13:53:35,895 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-26 13:53:35,898 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-26 13:53:35,898 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-26 13:53:35,901 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679838815900,5,FailOnTimeoutGroup] 2023-03-26 13:53:35,901 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679838815901,5,FailOnTimeoutGroup] 2023-03-26 13:53:35,901 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-26 13:53:35,901 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-26 13:53:35,903 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-26 13:53:35,903 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-26 13:53:35,912 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-26 13:53:35,914 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-26 13:53:35,914 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366 2023-03-26 13:53:35,921 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56573, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-26 13:53:35,928 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:53:35,932 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-26 13:53:35,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:35,936 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/info 2023-03-26 13:53:35,937 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-26 13:53:35,938 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:53:35,939 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-26 13:53:35,942 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/rep_barrier 2023-03-26 13:53:35,943 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-26 13:53:35,944 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:53:35,944 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-26 13:53:35,947 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/table 2023-03-26 13:53:35,947 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-26 13:53:35,948 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:53:35,950 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740 2023-03-26 13:53:35,951 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740 2023-03-26 13:53:35,955 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 13:53:35,956 DEBUG [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366 2023-03-26 13:53:35,956 DEBUG [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:40005 2023-03-26 13:53:35,956 DEBUG [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-26 13:53:35,959 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-26 13:53:35,963 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-26 13:53:35,963 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-26 13:53:35,964 DEBUG [RS:0;jenkins-hbase4:42517] zookeeper.ZKUtil(162): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:35,964 WARN [RS:0;jenkins-hbase4:42517] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-26 13:53:35,964 INFO [RS:0;jenkins-hbase4:42517] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-26 13:53:35,965 DEBUG [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1947): logDir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/WALs/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:35,965 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60455561, jitterRate=-0.09914194047451019}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 13:53:35,965 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-26 13:53:35,965 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-26 13:53:35,965 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-26 13:53:35,965 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-26 13:53:35,965 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-26 13:53:35,965 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-26 13:53:35,967 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-26 13:53:35,967 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,42517,1679838814721] 2023-03-26 13:53:35,967 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-26 13:53:35,974 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-26 13:53:35,974 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-26 13:53:35,977 DEBUG [RS:0;jenkins-hbase4:42517] zookeeper.ZKUtil(162): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:35,983 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-26 13:53:35,988 DEBUG [RS:0;jenkins-hbase4:42517] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-26 13:53:35,995 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-26 13:53:35,998 INFO [RS:0;jenkins-hbase4:42517] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-26 13:53:35,998 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-26 13:53:36,021 INFO [RS:0;jenkins-hbase4:42517] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-26 13:53:36,024 INFO [RS:0;jenkins-hbase4:42517] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-26 13:53:36,024 INFO [RS:0;jenkins-hbase4:42517] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-26 13:53:36,026 INFO [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-26 13:53:36,033 INFO [RS:0;jenkins-hbase4:42517] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-26 13:53:36,034 DEBUG [RS:0;jenkins-hbase4:42517] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 13:53:36,034 DEBUG [RS:0;jenkins-hbase4:42517] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 13:53:36,034 DEBUG [RS:0;jenkins-hbase4:42517] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 13:53:36,034 DEBUG [RS:0;jenkins-hbase4:42517] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 13:53:36,034 DEBUG [RS:0;jenkins-hbase4:42517] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 13:53:36,034 DEBUG [RS:0;jenkins-hbase4:42517] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-26 13:53:36,034 DEBUG [RS:0;jenkins-hbase4:42517] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 13:53:36,034 DEBUG [RS:0;jenkins-hbase4:42517] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 13:53:36,035 DEBUG [RS:0;jenkins-hbase4:42517] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 13:53:36,035 DEBUG [RS:0;jenkins-hbase4:42517] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 13:53:36,036 INFO [RS:0;jenkins-hbase4:42517] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-26 13:53:36,036 INFO [RS:0;jenkins-hbase4:42517] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-26 13:53:36,036 INFO [RS:0;jenkins-hbase4:42517] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-26 13:53:36,053 INFO [RS:0;jenkins-hbase4:42517] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-26 13:53:36,055 INFO [RS:0;jenkins-hbase4:42517] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,42517,1679838814721-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-26 13:53:36,070 INFO [RS:0;jenkins-hbase4:42517] regionserver.Replication(203): jenkins-hbase4.apache.org,42517,1679838814721 started 2023-03-26 13:53:36,070 INFO [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,42517,1679838814721, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:42517, sessionid=0x101211a03bf0001 2023-03-26 13:53:36,070 DEBUG [RS:0;jenkins-hbase4:42517] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-26 13:53:36,070 DEBUG [RS:0;jenkins-hbase4:42517] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:36,070 DEBUG [RS:0;jenkins-hbase4:42517] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42517,1679838814721' 2023-03-26 13:53:36,070 DEBUG [RS:0;jenkins-hbase4:42517] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:36,071 DEBUG [RS:0;jenkins-hbase4:42517] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:36,072 DEBUG [RS:0;jenkins-hbase4:42517] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-26 13:53:36,072 DEBUG [RS:0;jenkins-hbase4:42517] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-26 13:53:36,072 DEBUG [RS:0;jenkins-hbase4:42517] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:36,072 DEBUG [RS:0;jenkins-hbase4:42517] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,42517,1679838814721' 2023-03-26 13:53:36,072 DEBUG [RS:0;jenkins-hbase4:42517] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-26 13:53:36,072 DEBUG [RS:0;jenkins-hbase4:42517] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-26 13:53:36,073 DEBUG [RS:0;jenkins-hbase4:42517] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-26 13:53:36,073 INFO [RS:0;jenkins-hbase4:42517] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-26 13:53:36,073 INFO [RS:0;jenkins-hbase4:42517] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-26 13:53:36,150 DEBUG [jenkins-hbase4:45485] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-03-26 13:53:36,153 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42517,1679838814721, state=OPENING 2023-03-26 13:53:36,161 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-26 13:53:36,162 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 13:53:36,163 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-26 13:53:36,166 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:53:36,186 INFO [RS:0;jenkins-hbase4:42517] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42517%2C1679838814721, suffix=, logDir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/WALs/jenkins-hbase4.apache.org,42517,1679838814721, archiveDir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/oldWALs, maxLogs=32 2023-03-26 13:53:36,204 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42615,DS-ce026d9f-bfd8-44bb-9c87-03a1980b8fd8,DISK] 2023-03-26 13:53:36,207 INFO [RS:0;jenkins-hbase4:42517] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/WALs/jenkins-hbase4.apache.org,42517,1679838814721/jenkins-hbase4.apache.org%2C42517%2C1679838814721.1679838816189 2023-03-26 13:53:36,208 DEBUG [RS:0;jenkins-hbase4:42517] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42615,DS-ce026d9f-bfd8-44bb-9c87-03a1980b8fd8,DISK]] 2023-03-26 13:53:36,353 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:36,355 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-26 13:53:36,359 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39004, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-26 13:53:36,369 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-26 13:53:36,370 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-26 13:53:36,373 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C42517%2C1679838814721.meta, suffix=.meta, logDir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/WALs/jenkins-hbase4.apache.org,42517,1679838814721, archiveDir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/oldWALs, maxLogs=32 2023-03-26 13:53:36,391 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:42615,DS-ce026d9f-bfd8-44bb-9c87-03a1980b8fd8,DISK] 2023-03-26 13:53:36,395 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/WALs/jenkins-hbase4.apache.org,42517,1679838814721/jenkins-hbase4.apache.org%2C42517%2C1679838814721.meta.1679838816375.meta 2023-03-26 13:53:36,396 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:42615,DS-ce026d9f-bfd8-44bb-9c87-03a1980b8fd8,DISK]] 2023-03-26 13:53:36,396 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-26 13:53:36,398 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-26 13:53:36,420 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-26 13:53:36,424 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-26 13:53:36,431 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-26 13:53:36,431 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:53:36,431 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-26 13:53:36,431 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-26 13:53:36,434 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-26 13:53:36,436 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/info 2023-03-26 13:53:36,436 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/info 2023-03-26 13:53:36,436 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-26 13:53:36,437 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:53:36,437 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-26 13:53:36,439 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/rep_barrier 2023-03-26 13:53:36,439 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/rep_barrier 2023-03-26 13:53:36,439 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-26 13:53:36,440 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:53:36,440 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-26 13:53:36,442 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/table 2023-03-26 13:53:36,442 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/table 2023-03-26 13:53:36,442 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-26 13:53:36,443 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:53:36,445 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740 2023-03-26 13:53:36,447 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740 2023-03-26 13:53:36,451 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 13:53:36,453 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-26 13:53:36,455 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70389488, jitterRate=0.04888510704040527}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 13:53:36,455 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-26 13:53:36,464 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1679838816344 2023-03-26 13:53:36,482 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-26 13:53:36,483 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-26 13:53:36,483 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,42517,1679838814721, state=OPEN 2023-03-26 13:53:36,486 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-26 13:53:36,486 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-26 13:53:36,490 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-26 13:53:36,490 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,42517,1679838814721 in 320 msec 2023-03-26 13:53:36,496 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-26 13:53:36,496 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 508 msec 2023-03-26 13:53:36,501 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 736 msec 2023-03-26 13:53:36,502 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1679838816501, completionTime=-1 2023-03-26 13:53:36,502 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-03-26 13:53:36,502 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-03-26 13:53:36,563 DEBUG [hconnection-0xcc5547b-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:53:36,567 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39010, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:53:36,586 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-03-26 13:53:36,586 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1679838876586 2023-03-26 13:53:36,587 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1679838936587 2023-03-26 13:53:36,587 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 84 msec 2023-03-26 13:53:36,610 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45485,1679838813504-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-26 13:53:36,610 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45485,1679838813504-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-26 13:53:36,610 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45485,1679838813504-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-26 13:53:36,612 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:45485, period=300000, unit=MILLISECONDS is enabled. 2023-03-26 13:53:36,612 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-26 13:53:36,618 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-26 13:53:36,626 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-26 13:53:36,627 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-26 13:53:36,637 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-26 13:53:36,639 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-26 13:53:36,642 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-26 13:53:36,667 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/hbase/namespace/98e3213d37305f33190db3afbc13024b 2023-03-26 13:53:36,670 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/hbase/namespace/98e3213d37305f33190db3afbc13024b empty. 2023-03-26 13:53:36,671 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/hbase/namespace/98e3213d37305f33190db3afbc13024b 2023-03-26 13:53:36,671 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-26 13:53:37,096 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-26 13:53:37,099 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 98e3213d37305f33190db3afbc13024b, NAME => 'hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp 2023-03-26 13:53:37,510 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:53:37,511 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 98e3213d37305f33190db3afbc13024b, disabling compactions & flushes 2023-03-26 13:53:37,511 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b. 2023-03-26 13:53:37,511 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b. 2023-03-26 13:53:37,511 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b. after waiting 0 ms 2023-03-26 13:53:37,511 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b. 2023-03-26 13:53:37,511 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b. 2023-03-26 13:53:37,511 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 98e3213d37305f33190db3afbc13024b: 2023-03-26 13:53:37,516 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-26 13:53:37,532 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679838817520"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679838817520"}]},"ts":"1679838817520"} 2023-03-26 13:53:37,561 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-26 13:53:37,563 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-26 13:53:37,568 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838817564"}]},"ts":"1679838817564"} 2023-03-26 13:53:37,573 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-26 13:53:37,582 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=98e3213d37305f33190db3afbc13024b, ASSIGN}] 2023-03-26 13:53:37,585 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=98e3213d37305f33190db3afbc13024b, ASSIGN 2023-03-26 13:53:37,586 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=98e3213d37305f33190db3afbc13024b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42517,1679838814721; forceNewPlan=false, retain=false 2023-03-26 13:53:37,738 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=98e3213d37305f33190db3afbc13024b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:37,738 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679838817737"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838817737"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838817737"}]},"ts":"1679838817737"} 2023-03-26 13:53:37,743 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 98e3213d37305f33190db3afbc13024b, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:53:37,902 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b. 2023-03-26 13:53:37,903 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 98e3213d37305f33190db3afbc13024b, NAME => 'hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b.', STARTKEY => '', ENDKEY => ''} 2023-03-26 13:53:37,904 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 98e3213d37305f33190db3afbc13024b 2023-03-26 13:53:37,904 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:53:37,904 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 98e3213d37305f33190db3afbc13024b 2023-03-26 13:53:37,904 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 98e3213d37305f33190db3afbc13024b 2023-03-26 13:53:37,906 INFO [StoreOpener-98e3213d37305f33190db3afbc13024b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 98e3213d37305f33190db3afbc13024b 2023-03-26 13:53:37,909 DEBUG [StoreOpener-98e3213d37305f33190db3afbc13024b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/namespace/98e3213d37305f33190db3afbc13024b/info 2023-03-26 13:53:37,909 DEBUG [StoreOpener-98e3213d37305f33190db3afbc13024b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/namespace/98e3213d37305f33190db3afbc13024b/info 2023-03-26 13:53:37,909 INFO [StoreOpener-98e3213d37305f33190db3afbc13024b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 98e3213d37305f33190db3afbc13024b columnFamilyName info 2023-03-26 13:53:37,910 INFO [StoreOpener-98e3213d37305f33190db3afbc13024b-1] regionserver.HStore(310): Store=98e3213d37305f33190db3afbc13024b/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:53:37,912 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/namespace/98e3213d37305f33190db3afbc13024b 2023-03-26 13:53:37,913 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/namespace/98e3213d37305f33190db3afbc13024b 2023-03-26 13:53:37,917 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 98e3213d37305f33190db3afbc13024b 2023-03-26 13:53:37,921 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/namespace/98e3213d37305f33190db3afbc13024b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-26 13:53:37,922 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 98e3213d37305f33190db3afbc13024b; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59511330, jitterRate=-0.11321207880973816}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-26 13:53:37,922 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 98e3213d37305f33190db3afbc13024b: 2023-03-26 13:53:37,924 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b., pid=6, masterSystemTime=1679838817896 2023-03-26 13:53:37,928 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b. 2023-03-26 13:53:37,928 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b. 2023-03-26 13:53:37,929 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=98e3213d37305f33190db3afbc13024b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:37,930 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679838817929"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679838817929"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679838817929"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679838817929"}]},"ts":"1679838817929"} 2023-03-26 13:53:37,937 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-26 13:53:37,937 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 98e3213d37305f33190db3afbc13024b, server=jenkins-hbase4.apache.org,42517,1679838814721 in 191 msec 2023-03-26 13:53:37,941 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-26 13:53:37,941 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=98e3213d37305f33190db3afbc13024b, ASSIGN in 355 msec 2023-03-26 13:53:37,942 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-26 13:53:37,943 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838817942"}]},"ts":"1679838817942"} 2023-03-26 13:53:37,946 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-26 13:53:37,949 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-26 13:53:37,952 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3210 sec 2023-03-26 13:53:38,040 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-26 13:53:38,042 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-26 13:53:38,042 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 13:53:38,079 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-26 13:53:38,101 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-26 13:53:38,107 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 34 msec 2023-03-26 13:53:38,113 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-26 13:53:38,126 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-26 13:53:38,130 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 17 msec 2023-03-26 13:53:38,140 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-26 13:53:38,142 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-26 13:53:38,142 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.341sec 2023-03-26 13:53:38,144 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-26 13:53:38,146 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-26 13:53:38,146 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-26 13:53:38,147 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45485,1679838813504-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-26 13:53:38,148 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,45485,1679838813504-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-26 13:53:38,158 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-26 13:53:38,167 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x164139d4 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:53:38,172 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45b2cf6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:53:38,186 DEBUG [hconnection-0x1d2c4aea-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:53:38,197 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39014, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:53:38,208 INFO [Listener at localhost/41531] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,45485,1679838813504 2023-03-26 13:53:38,231 INFO [Listener at localhost/41531] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=208, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=340, ProcessCount=169, AvailableMemoryMB=9709 2023-03-26 13:53:38,249 DEBUG [Listener at localhost/41531] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-26 13:53:38,253 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35118, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-26 13:53:38,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-26 13:53:38,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 13:53:38,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-26 13:53:38,270 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-26 13:53:38,272 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-26 13:53:38,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-03-26 13:53:38,275 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:38,276 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8 empty. 2023-03-26 13:53:38,278 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:38,278 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 13:53:38,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-26 13:53:38,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-26 13:53:38,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-26 13:53:38,695 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-26 13:53:38,698 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => c54bfce2a8106a499411906befe41bd8, NAME => 'TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp 2023-03-26 13:53:38,710 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:53:38,710 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing c54bfce2a8106a499411906befe41bd8, disabling compactions & flushes 2023-03-26 13:53:38,711 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:38,711 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:38,711 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. after waiting 0 ms 2023-03-26 13:53:38,711 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:38,711 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:38,711 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:38,715 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-26 13:53:38,716 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838818716"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679838818716"}]},"ts":"1679838818716"} 2023-03-26 13:53:38,719 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-26 13:53:38,720 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-26 13:53:38,721 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838818721"}]},"ts":"1679838818721"} 2023-03-26 13:53:38,723 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-26 13:53:38,728 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c54bfce2a8106a499411906befe41bd8, ASSIGN}] 2023-03-26 13:53:38,730 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c54bfce2a8106a499411906befe41bd8, ASSIGN 2023-03-26 13:53:38,731 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c54bfce2a8106a499411906befe41bd8, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42517,1679838814721; forceNewPlan=false, retain=false 2023-03-26 13:53:38,882 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=c54bfce2a8106a499411906befe41bd8, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:38,883 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838818882"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838818882"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838818882"}]},"ts":"1679838818882"} 2023-03-26 13:53:38,886 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:53:38,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-26 13:53:39,045 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:39,046 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c54bfce2a8106a499411906befe41bd8, NAME => 'TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.', STARTKEY => '', ENDKEY => ''} 2023-03-26 13:53:39,046 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:39,046 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:53:39,046 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:39,046 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:39,048 INFO [StoreOpener-c54bfce2a8106a499411906befe41bd8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:39,050 DEBUG [StoreOpener-c54bfce2a8106a499411906befe41bd8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A 2023-03-26 13:53:39,050 DEBUG [StoreOpener-c54bfce2a8106a499411906befe41bd8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A 2023-03-26 13:53:39,052 INFO [StoreOpener-c54bfce2a8106a499411906befe41bd8-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:53:39,078 INFO [StoreOpener-c54bfce2a8106a499411906befe41bd8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c54bfce2a8106a499411906befe41bd8 columnFamilyName A 2023-03-26 13:53:39,079 INFO [StoreOpener-c54bfce2a8106a499411906befe41bd8-1] regionserver.HStore(310): Store=c54bfce2a8106a499411906befe41bd8/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:53:39,081 INFO [StoreOpener-c54bfce2a8106a499411906befe41bd8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:39,085 DEBUG [StoreOpener-c54bfce2a8106a499411906befe41bd8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B 2023-03-26 13:53:39,085 DEBUG [StoreOpener-c54bfce2a8106a499411906befe41bd8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B 2023-03-26 13:53:39,086 INFO [StoreOpener-c54bfce2a8106a499411906befe41bd8-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:53:39,086 INFO [StoreOpener-c54bfce2a8106a499411906befe41bd8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c54bfce2a8106a499411906befe41bd8 columnFamilyName B 2023-03-26 13:53:39,087 INFO [StoreOpener-c54bfce2a8106a499411906befe41bd8-1] regionserver.HStore(310): Store=c54bfce2a8106a499411906befe41bd8/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:53:39,087 INFO [StoreOpener-c54bfce2a8106a499411906befe41bd8-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:39,089 DEBUG [StoreOpener-c54bfce2a8106a499411906befe41bd8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C 2023-03-26 13:53:39,089 DEBUG [StoreOpener-c54bfce2a8106a499411906befe41bd8-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C 2023-03-26 13:53:39,089 INFO [StoreOpener-c54bfce2a8106a499411906befe41bd8-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:53:39,089 INFO [StoreOpener-c54bfce2a8106a499411906befe41bd8-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c54bfce2a8106a499411906befe41bd8 columnFamilyName C 2023-03-26 13:53:39,090 INFO [StoreOpener-c54bfce2a8106a499411906befe41bd8-1] regionserver.HStore(310): Store=c54bfce2a8106a499411906befe41bd8/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:53:39,091 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:39,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:39,093 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:39,096 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 13:53:39,099 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:39,102 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-26 13:53:39,102 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c54bfce2a8106a499411906befe41bd8; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67361308, jitterRate=0.0037617087364196777}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 13:53:39,103 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:39,104 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., pid=11, masterSystemTime=1679838819039 2023-03-26 13:53:39,106 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:39,106 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:39,107 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=c54bfce2a8106a499411906befe41bd8, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,108 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838819107"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679838819107"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679838819107"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679838819107"}]},"ts":"1679838819107"} 2023-03-26 13:53:39,113 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-03-26 13:53:39,113 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 in 224 msec 2023-03-26 13:53:39,116 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-03-26 13:53:39,117 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c54bfce2a8106a499411906befe41bd8, ASSIGN in 385 msec 2023-03-26 13:53:39,118 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-26 13:53:39,118 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838819118"}]},"ts":"1679838819118"} 2023-03-26 13:53:39,120 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-26 13:53:39,123 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-26 13:53:39,125 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 857 msec 2023-03-26 13:53:39,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-26 13:53:39,397 INFO [Listener at localhost/41531] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-03-26 13:53:39,402 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x23162110 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:53:39,409 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39bf63a2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:53:39,412 DEBUG [Listener at localhost/41531] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:53:39,415 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39024, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:53:39,419 DEBUG [Listener at localhost/41531] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-26 13:53:39,422 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35132, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-26 13:53:39,430 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x1e82daee to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:53:39,435 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@377933e4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:53:39,436 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x36dc7f7f to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:53:39,443 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@401974c7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:53:39,444 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x2e2005d0 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:53:39,448 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1ca4a062, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:53:39,448 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x79fee867 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:53:39,459 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ec07124, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:53:39,459 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x75f5a5e6 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:53:39,463 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51ef1151, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:53:39,464 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x64804459 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:53:39,468 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ae6c42b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:53:39,469 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x142b9819 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:53:39,513 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@726fb449, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:53:39,513 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x6fcbe28d to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:53:39,525 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6089dec1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:53:39,525 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x3c9ded19 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:53:39,532 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@c72af8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:53:39,541 DEBUG [hconnection-0x3fdf621b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:53:39,542 DEBUG [hconnection-0x3685aa1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:53:39,543 DEBUG [hconnection-0xb79970f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:53:39,543 DEBUG [hconnection-0x65e031ee-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:53:39,544 DEBUG [hconnection-0x1b518f7a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:53:39,544 DEBUG [hconnection-0x43a25080-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:53:39,545 DEBUG [hconnection-0x1a5dd2ff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:53:39,550 DEBUG [hconnection-0x65d3daa5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:53:39,550 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39034, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:53:39,551 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39044, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:53:39,552 DEBUG [hconnection-0x20746307-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:53:39,558 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39056, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:53:39,558 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39046, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:53:39,560 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39068, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:53:39,560 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39074, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:53:39,561 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39064, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:53:39,560 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39054, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:53:39,569 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39082, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:53:39,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:39,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:39,604 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:39,604 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:39,605 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:39,605 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:39,606 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:39,606 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:39,608 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:39,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:39,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:39,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:39,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:39,610 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:39,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:39,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:39,613 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:39,613 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,613 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:39,618 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:39,618 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:39,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:39,634 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:39,634 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 13:53:39,638 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:39,639 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:39,639 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:39,644 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:39,644 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:39,648 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:39,649 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:39,649 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:39,649 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,649 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:39,649 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:39,654 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,654 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:39,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:39,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:39,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:39,655 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:39,655 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:39,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:39,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:39,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:39,657 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:39,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:39,658 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:39,658 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:39,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d4a2ba[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:39,658 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:39,658 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:39,659 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:39,659 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:39,659 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:39,662 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:39,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:39,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:39,663 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,663 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:39,663 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:39,663 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:39,663 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:39,668 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:39,668 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:39,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:39,668 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:39,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:39,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:39,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:39,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:39,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:39,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:39,674 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:39,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:39,674 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:39,675 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:39,675 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:39,675 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:39,678 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:39,678 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:39,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:39,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:39,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:39,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:39,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:39,684 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:39,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:39,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:39,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:39,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:39,687 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:39,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:39,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:39,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,691 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:39,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:39,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,710 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:39,710 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:39,710 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:39,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:39,710 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:39,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:39,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,711 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:39,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:39,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:39,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:39,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:39,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:39,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:39,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:39,713 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:39,713 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:39,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:39,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838879736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:39,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838879738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:39,758 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:39,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838879739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838879744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:39,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838879745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,778 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/cfe19306a224425eac61309608def4aa 2023-03-26 13:53:39,814 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:39,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:39,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:39,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:39,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:39,848 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:39,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:39,852 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:39,852 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:39,853 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:39,853 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:39,861 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:39,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:39,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:39,862 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:39,862 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,863 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:39,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:39,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:39,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:39,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:39,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:39,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:39,868 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:39,868 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:39,872 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:39,873 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:39,873 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:39,874 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:39,874 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:39,874 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:39,874 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:39,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:39,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:39,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:39,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:39,890 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:39,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:39,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:39,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:39,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:39,895 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:39,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:39,899 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:39,899 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:39,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:39,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838879899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:39,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838879900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:39,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838879901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:39,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838879901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:39,904 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:39,904 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:39,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@170efc7[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:39,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838879901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,906 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:39,908 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:39,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:39,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:39,908 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:39,908 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:39,908 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:39,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:39,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:39,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:39,915 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,920 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:39,920 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:39,920 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:39,920 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:39,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:39,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:39,921 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:39,922 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/e58a1937955b45b283728b077f1880ff 2023-03-26 13:53:39,927 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:39,927 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:39,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:39,927 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:39,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:39,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:39,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:39,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:39,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:39,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:39,928 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:39,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:39,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:39,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:39,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:39,932 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:39,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:39,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,946 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:39,946 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:39,946 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:39,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:39,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:39,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:39,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:39,947 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:39,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:39,947 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:39,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:39,948 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,948 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:39,948 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:39,948 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:39,948 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:39,948 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:39,948 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,047 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:40,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:40,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:40,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:40,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:40,065 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:40,065 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:40,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:40,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:40,067 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,067 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:40,069 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,069 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:40,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:40,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:40,069 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,069 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:40,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:40,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:40,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:40,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:40,071 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:40,071 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:40,072 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:40,072 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:40,076 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:40,077 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:40,077 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:40,077 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:40,077 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:40,077 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:40,077 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:40,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:40,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:40,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:40,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:40,082 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,082 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:40,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:40,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:40,083 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,083 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:40,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:40,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:40,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:40,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:40,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53f2aa01[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:40,085 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:40,085 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,088 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,088 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:40,088 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:40,088 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:40,089 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,089 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:40,091 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,091 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:40,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,091 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:40,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:40,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:40,091 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:40,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:40,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:40,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:40,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:40,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:40,099 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:40,099 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:40,099 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:40,099 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:40,099 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:40,101 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,101 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:40,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,101 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:40,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:40,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:40,102 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:40,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:40,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:40,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:40,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:40,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838880106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:40,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:40,108 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838880106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:40,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838880107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:40,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:40,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:40,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838880109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838880110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,115 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:40,115 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:40,115 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:40,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:40,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:40,115 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:40,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:40,116 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:40,116 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:40,117 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,117 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,117 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,117 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,117 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:40,117 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:40,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:40,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:40,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:40,217 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:40,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:40,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:40,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:40,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:40,228 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:40,228 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:40,232 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:40,232 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:40,233 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,233 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:40,236 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:40,236 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:40,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:40,237 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,237 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:40,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:40,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:40,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:40,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:40,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:40,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:40,239 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:40,239 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:40,240 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:40,240 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:40,240 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:40,240 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:40,240 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:40,240 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:40,241 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:40,244 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,244 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:40,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:40,245 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,245 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:40,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:40,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:40,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:40,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:40,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:40,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a7d520a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:40,248 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:40,248 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,250 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,250 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,250 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:40,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,250 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:40,250 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:40,250 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:40,257 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:40,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:40,257 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:40,257 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:40,257 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:40,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:40,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:40,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:40,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:40,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:40,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:40,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:40,263 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:40,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:40,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:40,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:40,264 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:40,268 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,268 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:40,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:40,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,268 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:40,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:40,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:40,270 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:40,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:40,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:40,272 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:40,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:40,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:40,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:40,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:40,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:40,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:40,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:40,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:40,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:40,281 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:40,281 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:40,281 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:40,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:40,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:40,382 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:40,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:40,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:40,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:40,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:40,398 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:40,398 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:40,400 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/2c403239205c479c9de92d8d19e3eafc 2023-03-26 13:53:40,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:40,412 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/cfe19306a224425eac61309608def4aa as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/cfe19306a224425eac61309608def4aa 2023-03-26 13:53:40,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838880412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:40,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838880412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:40,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838880413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:40,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838880416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:40,429 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:40,429 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:40,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838880418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,430 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,430 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:40,432 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/cfe19306a224425eac61309608def4aa, entries=150, sequenceid=14, filesize=11.6 K 2023-03-26 13:53:40,433 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:40,433 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:40,434 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,434 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:40,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:40,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:40,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:40,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:40,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:40,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:40,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:40,436 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:40,437 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:40,437 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:40,438 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:40,438 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:40,438 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:40,438 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:40,438 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:40,438 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:40,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:40,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:40,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:40,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:40,442 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/e58a1937955b45b283728b077f1880ff as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e58a1937955b45b283728b077f1880ff 2023-03-26 13:53:40,444 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:40,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:40,444 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:40,444 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,445 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:40,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:40,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:40,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:40,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:40,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b9dec8d[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:40,446 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:40,446 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,451 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,453 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:40,453 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:40,453 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,453 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:40,453 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:40,455 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,455 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:40,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,455 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:40,455 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:40,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:40,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:40,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:40,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:40,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:40,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:40,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:40,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:40,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:40,460 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:40,460 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:40,460 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:40,460 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:40,460 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:40,462 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,462 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:40,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:40,462 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:40,463 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:40,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:40,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:40,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:40,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:40,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:40,468 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:40,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:40,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,477 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:40,477 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:40,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:40,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:40,479 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,479 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:40,479 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:40,479 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:40,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:40,479 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,480 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:40,480 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:40,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:40,479 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:40,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:40,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:40,480 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e58a1937955b45b283728b077f1880ff, entries=150, sequenceid=14, filesize=11.6 K 2023-03-26 13:53:40,491 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/2c403239205c479c9de92d8d19e3eafc as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/2c403239205c479c9de92d8d19e3eafc 2023-03-26 13:53:40,509 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/2c403239205c479c9de92d8d19e3eafc, entries=150, sequenceid=14, filesize=11.6 K 2023-03-26 13:53:40,519 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for c54bfce2a8106a499411906befe41bd8 in 885ms, sequenceid=14, compaction requested=false 2023-03-26 13:53:40,521 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-26 13:53:40,524 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:40,580 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:40,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:40,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:40,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:40,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:40,591 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:40,591 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:40,593 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:40,593 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:40,593 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,593 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:40,596 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:40,596 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:40,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:40,596 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,597 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:40,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:40,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:40,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:40,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:40,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:40,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:40,606 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:40,606 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:40,608 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:40,608 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:40,608 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:40,609 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:40,609 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:40,609 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-26 13:53:40,609 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:40,610 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:40,610 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:40,610 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:40,610 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:40,610 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:40,729 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/33836bb6787c43c2969df6e890970e8b 2023-03-26 13:53:40,801 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/3896817d58af4e3893a8db480a9229f6 2023-03-26 13:53:40,870 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/e21a1bb11ce84ca2831379da85896b3b 2023-03-26 13:53:40,881 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/33836bb6787c43c2969df6e890970e8b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/33836bb6787c43c2969df6e890970e8b 2023-03-26 13:53:40,893 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/33836bb6787c43c2969df6e890970e8b, entries=150, sequenceid=34, filesize=11.6 K 2023-03-26 13:53:40,897 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/3896817d58af4e3893a8db480a9229f6 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/3896817d58af4e3893a8db480a9229f6 2023-03-26 13:53:40,910 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/3896817d58af4e3893a8db480a9229f6, entries=150, sequenceid=34, filesize=11.6 K 2023-03-26 13:53:40,912 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/e21a1bb11ce84ca2831379da85896b3b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e21a1bb11ce84ca2831379da85896b3b 2023-03-26 13:53:40,928 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:40,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:40,947 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e21a1bb11ce84ca2831379da85896b3b, entries=150, sequenceid=34, filesize=11.6 K 2023-03-26 13:53:40,987 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=13.42 KB/13740 for c54bfce2a8106a499411906befe41bd8 in 378ms, sequenceid=34, compaction requested=false 2023-03-26 13:53:40,988 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:40,988 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:40,988 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:40,989 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:40,989 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,989 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:40,989 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:40,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:40,994 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:53:40,994 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:40,995 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:40,995 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:40,995 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:40,995 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:40,995 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:40,996 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:40,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:40,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:40,997 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:40,998 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:40,998 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:40,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:40,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:41,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:41,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d153cee[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:41,001 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:41,001 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,003 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,003 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:41,003 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:41,003 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:41,008 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:41,008 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:41,008 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:41,013 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,013 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:41,013 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:41,014 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:41,014 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:41,015 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:41,015 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:41,015 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:41,017 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,017 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:41,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:41,017 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:41,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:41,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:41,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:41,018 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:41,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,020 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:41,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:41,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:41,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:41,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:41,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:41,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:41,031 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:41,031 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:41,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,031 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,031 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,031 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,031 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,031 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:41,031 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:41,031 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:41,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:41,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838881066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838881070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,092 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838821036, max timestamp=1679838821075Num uniques 859; 2023-03-26 13:53:41,093 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:53:41,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838821036, max timestamp=1679838821076Num uniques 859; 2023-03-26 13:53:41,094 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:53:41,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679838821036, max timestamp=1679838821076Num uniques 859; 2023-03-26 13:53:41,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-26 13:53:41,132 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:41,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:41,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:41,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:41,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:41,140 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:41,140 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:41,141 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:41,141 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:41,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,143 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:41,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:41,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:41,148 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,149 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,149 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:41,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:41,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:41,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:41,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:41,150 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:41,150 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:41,152 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:41,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:41,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:41,153 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:41,153 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:41,153 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:41,153 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:41,155 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:41,155 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:41,155 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,155 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:41,155 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:41,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:41,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:41,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838881148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:41,159 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:41,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,160 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838881156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,160 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:41,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:41,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:41,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b7125ef[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:41,162 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:41,162 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,164 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,165 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,165 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:41,165 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:41,165 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:41,165 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:41,168 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838881158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,169 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,169 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:41,169 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:41,169 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:41,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:41,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:41,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:41,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:41,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:41,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:41,176 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:41,176 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:41,176 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:41,176 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:41,176 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:41,179 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,179 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,179 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:41,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:41,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:41,182 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:41,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,185 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,191 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:41,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:41,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:41,192 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:41,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:41,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:41,193 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,193 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,194 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:41,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:41,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:41,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:41,196 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,196 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,196 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,196 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:41,196 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:41,196 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,197 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,197 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838881191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838881197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838881262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838881267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838881271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,297 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:41,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:41,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:41,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:41,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:41,312 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:41,312 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:41,313 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:41,313 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:41,314 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,314 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,316 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:41,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:41,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:41,317 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,317 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,318 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:41,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:41,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:41,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:41,320 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:41,320 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:41,321 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:41,328 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:41,328 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:41,336 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:41,336 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:41,336 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:41,336 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:41,337 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:41,337 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:41,337 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,337 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:41,337 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:41,340 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,340 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:41,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:41,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:41,340 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,341 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:41,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:41,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b1ccb89[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:41,343 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:41,343 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,344 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,345 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:41,345 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:41,345 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:41,346 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,346 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:41,348 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:41,348 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:41,348 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:41,348 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:41,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:41,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:41,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:41,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:41,357 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:41,357 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:41,357 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:41,357 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:41,357 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:41,362 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,362 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:41,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,363 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:41,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:41,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:41,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:41,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:41,368 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:41,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:41,371 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,380 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,380 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:41,380 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:41,380 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:41,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:41,380 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:41,380 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:41,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:41,382 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,382 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,382 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,382 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,382 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:41,383 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:41,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:41,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:41,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838881400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,403 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838881401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,466 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=44 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/df64391b6e3c42aca27635cae2019273 2023-03-26 13:53:41,483 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:41,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:41,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838881482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838881486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838881485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:41,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:41,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:41,529 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:41,529 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:41,532 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:41,535 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:41,536 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,536 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,539 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,539 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:41,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:41,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:41,540 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,540 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:41,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:41,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:41,556 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:41,556 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:41,556 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:41,556 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:41,557 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:41,560 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:41,560 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:41,561 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=44 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/06c6a4d76392467b9da4fdf6e2b90367 2023-03-26 13:53:41,564 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:41,564 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:41,564 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:41,565 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:41,568 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:41,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:41,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:41,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:41,571 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:41,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:41,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:41,572 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,573 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:41,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:41,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:41,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:41,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f41a7e3[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:41,581 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:41,581 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,583 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,583 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:41,583 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:41,583 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:41,583 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,583 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:41,585 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,585 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:41,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:41,585 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:41,585 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:41,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:41,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:41,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:41,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:41,597 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:41,597 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:41,597 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:41,597 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:41,597 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:41,608 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,609 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:41,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,608 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:41,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:41,612 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:41,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:41,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:41,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:41,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,614 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:41,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:41,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:41,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:41,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:41,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:41,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:41,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:41,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:41,622 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,622 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,622 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,622 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,623 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,623 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:41,623 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,623 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:41,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:41,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:41,624 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:41,624 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:41,638 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=44 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/9ab0325e4a5441859cc3722b76a68ac4 2023-03-26 13:53:41,649 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/df64391b6e3c42aca27635cae2019273 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/df64391b6e3c42aca27635cae2019273 2023-03-26 13:53:41,667 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/df64391b6e3c42aca27635cae2019273, entries=200, sequenceid=44, filesize=13.9 K 2023-03-26 13:53:41,669 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/06c6a4d76392467b9da4fdf6e2b90367 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/06c6a4d76392467b9da4fdf6e2b90367 2023-03-26 13:53:41,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/06c6a4d76392467b9da4fdf6e2b90367, entries=150, sequenceid=44, filesize=11.6 K 2023-03-26 13:53:41,696 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/9ab0325e4a5441859cc3722b76a68ac4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/9ab0325e4a5441859cc3722b76a68ac4 2023-03-26 13:53:41,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838881707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838881708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,717 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/9ab0325e4a5441859cc3722b76a68ac4, entries=150, sequenceid=44, filesize=11.6 K 2023-03-26 13:53:41,720 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.55 KB/134705 for c54bfce2a8106a499411906befe41bd8 in 726ms, sequenceid=44, compaction requested=true 2023-03-26 13:53:41,720 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:41,724 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:41,724 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:41,725 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:41,725 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:41,725 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:41,725 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:41,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:41,732 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:41,735 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:41,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:41,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:41,739 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:41,739 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:41,739 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:41,739 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:41,740 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:41,740 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,740 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,740 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/B is initiating minor compaction (all files) 2023-03-26 13:53:41,740 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/B in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:41,741 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/A is initiating minor compaction (all files) 2023-03-26 13:53:41,741 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e58a1937955b45b283728b077f1880ff, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/3896817d58af4e3893a8db480a9229f6, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/06c6a4d76392467b9da4fdf6e2b90367] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=34.8 K 2023-03-26 13:53:41,741 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/A in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:41,741 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/cfe19306a224425eac61309608def4aa, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/33836bb6787c43c2969df6e890970e8b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/df64391b6e3c42aca27635cae2019273] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=37.1 K 2023-03-26 13:53:41,742 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:41,742 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,749 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e58a1937955b45b283728b077f1880ff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679838819583 2023-03-26 13:53:41,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:41,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:41,749 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,750 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:41,750 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting cfe19306a224425eac61309608def4aa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679838819583 2023-03-26 13:53:41,750 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 3896817d58af4e3893a8db480a9229f6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1679838819721 2023-03-26 13:53:41,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:41,751 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 33836bb6787c43c2969df6e890970e8b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1679838819721 2023-03-26 13:53:41,751 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 06c6a4d76392467b9da4fdf6e2b90367, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=44, earliestPutTs=1679838820934 2023-03-26 13:53:41,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:41,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:41,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:41,752 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting df64391b6e3c42aca27635cae2019273, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=44, earliestPutTs=1679838820934 2023-03-26 13:53:41,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:41,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:41,755 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:41,755 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:41,758 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:41,758 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:41,758 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:41,758 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:41,758 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:41,758 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=131.55 KB heapSize=374.30 KB 2023-03-26 13:53:41,759 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:41,759 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:41,759 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:41,759 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:41,760 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:41,760 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:41,794 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#B#compaction#10 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:41,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:41,797 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:41,831 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#A#compaction#11 average throughput is 0.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:41,860 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/eef618848cec41c994118d92ae77ef18 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/eef618848cec41c994118d92ae77ef18 2023-03-26 13:53:41,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838881853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838881860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838881860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,885 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/B of c54bfce2a8106a499411906befe41bd8 into eef618848cec41c994118d92ae77ef18(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:41,885 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:41,885 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/B, priority=13, startTime=1679838821725; duration=0sec 2023-03-26 13:53:41,886 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:41,886 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:41,890 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:41,890 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/C is initiating minor compaction (all files) 2023-03-26 13:53:41,890 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/C in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:41,890 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/2c403239205c479c9de92d8d19e3eafc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e21a1bb11ce84ca2831379da85896b3b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/9ab0325e4a5441859cc3722b76a68ac4] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=34.8 K 2023-03-26 13:53:41,891 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 2c403239205c479c9de92d8d19e3eafc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679838819583 2023-03-26 13:53:41,892 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e21a1bb11ce84ca2831379da85896b3b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1679838819721 2023-03-26 13:53:41,893 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 9ab0325e4a5441859cc3722b76a68ac4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=44, earliestPutTs=1679838820934 2023-03-26 13:53:41,906 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#C#compaction#12 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:41,961 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/d43ee3f0619a449aa6158b78965c6bbb as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/d43ee3f0619a449aa6158b78965c6bbb 2023-03-26 13:53:41,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838881971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838881971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:41,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838881972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:41,990 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/C of c54bfce2a8106a499411906befe41bd8 into d43ee3f0619a449aa6158b78965c6bbb(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:41,990 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:41,990 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/C, priority=13, startTime=1679838821725; duration=0sec 2023-03-26 13:53:41,990 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:42,055 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-26 13:53:42,170 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-26 13:53:42,171 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-26 13:53:42,187 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:42,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838882185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:42,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838882185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,189 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:42,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838882189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:42,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838882210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:42,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838882214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,227 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/242e392f053041a9b9c3ea52eeccf17e 2023-03-26 13:53:42,255 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/98e3dbc4641b436eb7049f32d6de1ec6 2023-03-26 13:53:42,277 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/99e1618bda1146a28a8656b8a9073b55 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/99e1618bda1146a28a8656b8a9073b55 2023-03-26 13:53:42,300 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/A of c54bfce2a8106a499411906befe41bd8 into 99e1618bda1146a28a8656b8a9073b55(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:42,300 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:42,300 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/A, priority=13, startTime=1679838821724; duration=0sec 2023-03-26 13:53:42,300 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:42,317 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=80 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/bfdbe64ec0da4223aeec5fcd8cb8dbf2 2023-03-26 13:53:42,338 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/242e392f053041a9b9c3ea52eeccf17e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/242e392f053041a9b9c3ea52eeccf17e 2023-03-26 13:53:42,378 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/242e392f053041a9b9c3ea52eeccf17e, entries=150, sequenceid=80, filesize=11.6 K 2023-03-26 13:53:42,382 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/98e3dbc4641b436eb7049f32d6de1ec6 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/98e3dbc4641b436eb7049f32d6de1ec6 2023-03-26 13:53:42,393 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/98e3dbc4641b436eb7049f32d6de1ec6, entries=150, sequenceid=80, filesize=11.6 K 2023-03-26 13:53:42,395 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/bfdbe64ec0da4223aeec5fcd8cb8dbf2 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/bfdbe64ec0da4223aeec5fcd8cb8dbf2 2023-03-26 13:53:42,410 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/bfdbe64ec0da4223aeec5fcd8cb8dbf2, entries=150, sequenceid=80, filesize=11.6 K 2023-03-26 13:53:42,413 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.55 KB/134705, heapSize ~374.26 KB/383240, currentSize=46.96 KB/48090 for c54bfce2a8106a499411906befe41bd8 in 654ms, sequenceid=80, compaction requested=false 2023-03-26 13:53:42,413 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:42,413 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:42,413 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:42,416 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:42,416 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,417 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:42,417 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:42,419 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,419 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:42,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:42,419 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,419 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:42,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:42,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:42,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:42,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:42,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bd71ca9[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:42,422 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:42,422 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,425 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,425 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:42,425 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:42,425 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:42,426 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,426 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:42,427 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:42,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:42,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:42,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:42,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:42,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:42,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:42,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:42,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:42,434 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:42,434 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:42,434 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:42,434 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:42,434 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:42,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:42,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:42,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:42,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:42,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:42,440 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:42,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:42,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,443 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:42,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,447 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:42,447 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:42,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:42,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:42,447 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,448 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:42,448 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:42,448 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,448 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,448 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,448 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:42,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:42,447 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:42,448 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:42,448 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:42,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:42,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:42,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:42,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:42,494 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 13:53:42,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:42,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:42,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:42,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:42,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:42,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:42,529 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/31ed28ae2c7b4af29ff994fd049eb89b 2023-03-26 13:53:42,548 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:42,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:42,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:42,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:42,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838882553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:42,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838882557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:42,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838882557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:42,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:42,562 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:42,562 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:42,563 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:42,563 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:42,566 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,566 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:42,572 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:42,572 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:42,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:42,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:42,573 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,573 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:42,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:42,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:42,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:42,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:42,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:42,575 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:42,575 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:42,575 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:42,576 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:42,580 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:42,580 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:42,580 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:42,580 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:42,580 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:42,580 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:42,580 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:42,580 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,580 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:42,580 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:42,586 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,586 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,586 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:42,586 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:42,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:42,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:42,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:42,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:42,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:42,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@490b1661[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:42,594 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:42,594 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,598 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,598 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:42,598 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:42,598 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:42,599 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,599 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:42,600 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,600 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:42,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:42,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:42,600 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:42,601 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:42,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:42,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:42,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:42,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:42,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:42,606 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:42,606 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:42,606 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:42,606 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:42,606 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:42,614 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,614 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:42,614 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:42,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:42,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:42,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:42,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:42,618 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:42,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,621 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,625 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:42,625 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:42,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:42,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:42,625 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,625 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:42,625 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:42,625 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:42,625 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,625 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:42,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:42,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:42,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:42,627 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:42,627 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:42,626 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,627 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,662 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:42,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838882660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:42,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838882661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:42,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838882661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,727 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:42,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:42,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:42,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:42,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:42,740 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:42,740 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:42,741 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:42,741 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:42,742 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,742 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:42,744 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:42,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:42,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:42,745 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,746 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,746 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:42,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:42,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:42,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:42,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:42,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:42,752 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:42,752 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:42,756 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:42,756 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:42,756 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:42,756 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:42,757 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:42,757 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:42,757 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:42,757 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:42,757 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:42,757 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,757 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:42,757 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:42,763 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,763 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:42,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:42,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:42,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:42,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,766 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,767 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:42,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:42,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:42,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2426cc4a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:42,769 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:42,769 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,772 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:42,772 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:42,772 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:42,772 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,772 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:42,774 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:42,774 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:42,774 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:42,774 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:42,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:42,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:42,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:42,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:42,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:42,780 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:42,780 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:42,780 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:42,780 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:42,780 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:42,783 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,783 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:42,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,783 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:42,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:42,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:42,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:42,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:42,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,784 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:42,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,786 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:42,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,793 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:42,793 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:42,793 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:42,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:42,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:42,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:42,794 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:42,794 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:42,794 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:42,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:42,794 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,797 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,797 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,797 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,795 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:42,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:42,797 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:42,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:42,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838882866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:42,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838882868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:42,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838882869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,897 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:42,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:42,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:42,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:42,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:42,903 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:42,903 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:42,904 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:42,904 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:42,905 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,905 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:42,906 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,906 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:42,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:42,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:42,909 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,909 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:42,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:42,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:42,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:42,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:42,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:42,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:42,911 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:42,911 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:42,912 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:42,914 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:42,914 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:42,915 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:42,915 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:42,915 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:42,915 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:42,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:42,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:42,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:42,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:42,917 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,917 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:42,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:42,917 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,917 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:42,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:42,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:42,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:42,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:42,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67a1992[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:42,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:42,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,923 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:42,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:42,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:42,927 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:42,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:42,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:42,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:42,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:42,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:42,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:42,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:42,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:42,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:42,935 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:42,935 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:42,935 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:42,935 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:42,935 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:42,937 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,937 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,937 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:42,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,938 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,940 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:42,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:42,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:42,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:42,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:42,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:42,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:42,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:42,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:42,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:42,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:42,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:42,945 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:42,946 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:42,947 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:42,987 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/51a3643804da423b90a756bcac5a73a6 2023-03-26 13:53:43,047 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:43,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:43,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:43,057 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=95 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/0f34332073924a738a8480807e4628c7 2023-03-26 13:53:43,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:43,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:43,059 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:43,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:43,060 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:43,060 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:43,062 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,062 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:43,064 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:43,064 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:43,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:43,065 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,065 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:43,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:43,067 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:43,070 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/31ed28ae2c7b4af29ff994fd049eb89b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/31ed28ae2c7b4af29ff994fd049eb89b 2023-03-26 13:53:43,079 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/31ed28ae2c7b4af29ff994fd049eb89b, entries=150, sequenceid=95, filesize=11.6 K 2023-03-26 13:53:43,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:43,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:43,081 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:43,081 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/51a3643804da423b90a756bcac5a73a6 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/51a3643804da423b90a756bcac5a73a6 2023-03-26 13:53:43,081 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:43,082 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:43,082 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:43,082 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:43,083 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:43,083 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:43,083 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:43,083 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:43,083 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:43,083 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:43,083 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,083 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:43,083 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:43,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,086 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:43,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:43,087 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:43,087 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:43,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:43,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:43,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:43,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:43,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75bb6fa0[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:43,089 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:43,089 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:43,091 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,091 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:43,092 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:43,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:43,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:43,092 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:43,092 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:43,092 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:43,092 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/51a3643804da423b90a756bcac5a73a6, entries=150, sequenceid=95, filesize=11.6 K 2023-03-26 13:53:43,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/0f34332073924a738a8480807e4628c7 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/0f34332073924a738a8480807e4628c7 2023-03-26 13:53:43,094 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,094 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:43,094 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:43,094 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:43,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:43,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:43,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:43,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:43,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:43,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:43,103 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:43,103 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:43,103 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:43,103 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:43,103 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:43,105 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,105 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:43,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,105 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:43,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:43,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,107 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:43,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:43,108 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/0f34332073924a738a8480807e4628c7, entries=150, sequenceid=95, filesize=11.6 K 2023-03-26 13:53:43,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:43,110 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:43,115 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c54bfce2a8106a499411906befe41bd8 in 620ms, sequenceid=95, compaction requested=true 2023-03-26 13:53:43,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:43,115 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:43,116 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:43,116 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:43,116 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:43,117 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:43,117 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:43,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,118 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:43,119 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/A is initiating minor compaction (all files) 2023-03-26 13:53:43,119 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/A in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:43,119 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/99e1618bda1146a28a8656b8a9073b55, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/242e392f053041a9b9c3ea52eeccf17e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/31ed28ae2c7b4af29ff994fd049eb89b] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=34.9 K 2023-03-26 13:53:43,119 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:43,119 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/B is initiating minor compaction (all files) 2023-03-26 13:53:43,119 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/B in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:43,119 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/eef618848cec41c994118d92ae77ef18, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/98e3dbc4641b436eb7049f32d6de1ec6, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/51a3643804da423b90a756bcac5a73a6] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=34.9 K 2023-03-26 13:53:43,120 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 99e1618bda1146a28a8656b8a9073b55, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=44, earliestPutTs=1679838820934 2023-03-26 13:53:43,125 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:43,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:43,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:43,127 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,127 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:43,127 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:43,127 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,127 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:43,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:43,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:43,127 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:43,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:43,127 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:43,129 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting eef618848cec41c994118d92ae77ef18, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=44, earliestPutTs=1679838820934 2023-03-26 13:53:43,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:43,129 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:43,129 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,129 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:43,129 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:43,130 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 98e3dbc4641b436eb7049f32d6de1ec6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1679838821127 2023-03-26 13:53:43,130 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 51a3643804da423b90a756bcac5a73a6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1679838821849 2023-03-26 13:53:43,133 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 242e392f053041a9b9c3ea52eeccf17e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1679838821127 2023-03-26 13:53:43,135 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 31ed28ae2c7b4af29ff994fd049eb89b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1679838821849 2023-03-26 13:53:43,149 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#B#compaction#18 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:43,153 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#A#compaction#19 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:43,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:43,188 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=127.47 KB heapSize=379.97 KB 2023-03-26 13:53:43,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838822535, max timestamp=1679838823177Num uniques 859; 2023-03-26 13:53:43,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:53:43,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838822535, max timestamp=1679838823179Num uniques 859; 2023-03-26 13:53:43,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:53:43,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:43,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:43,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:43,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:43,196 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:43,196 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:43,225 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/5072ab0a565d49679f215d5f5d19ac22 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/5072ab0a565d49679f215d5f5d19ac22 2023-03-26 13:53:43,230 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:43,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:43,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:43,241 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/A of c54bfce2a8106a499411906befe41bd8 into 5072ab0a565d49679f215d5f5d19ac22(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:43,241 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:43,241 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/A, priority=13, startTime=1679838823115; duration=0sec 2023-03-26 13:53:43,241 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:43,242 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:43,243 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:43,243 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/C is initiating minor compaction (all files) 2023-03-26 13:53:43,243 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/C in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:43,244 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/d43ee3f0619a449aa6158b78965c6bbb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/bfdbe64ec0da4223aeec5fcd8cb8dbf2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/0f34332073924a738a8480807e4628c7] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=34.9 K 2023-03-26 13:53:43,244 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting d43ee3f0619a449aa6158b78965c6bbb, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=44, earliestPutTs=1679838820934 2023-03-26 13:53:43,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:43,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838883231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,245 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting bfdbe64ec0da4223aeec5fcd8cb8dbf2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=80, earliestPutTs=1679838821127 2023-03-26 13:53:43,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:43,248 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:43,248 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:43,249 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:43,249 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:43,250 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 0f34332073924a738a8480807e4628c7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1679838821849 2023-03-26 13:53:43,250 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,250 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/1618bf3e7fc144bfa7e5c408f5165d6a 2023-03-26 13:53:43,250 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:43,254 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,254 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:43,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:43,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:43,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,256 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838883240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:43,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838883245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,256 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:43,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838883241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:43,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:43,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838883245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:43,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:43,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:43,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:43,289 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:43,289 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:43,289 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:43,291 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#C#compaction#22 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:43,294 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:43,294 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:43,294 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:43,294 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:43,294 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:43,294 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:43,294 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,294 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:43,294 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:43,298 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:43,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:43,298 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:43,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:43,299 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:43,299 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:43,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:43,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:43,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:43,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:43,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6492c768[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:43,318 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:43,318 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:43,318 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/88344d7c986741149c0da35967a9faed 2023-03-26 13:53:43,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:43,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:43,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:43,323 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:43,323 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:43,323 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:43,328 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,328 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:43,330 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:43,330 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,330 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:43,331 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:43,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:43,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:43,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:43,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:43,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:43,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:43,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:43,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:43,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:43,342 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:43,342 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:43,343 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:43,343 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:43,343 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:43,343 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/d11453e7f7c24115a38969285da124d3 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/d11453e7f7c24115a38969285da124d3 2023-03-26 13:53:43,350 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,350 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:43,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:43,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:43,351 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,351 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:43,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:43,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:43,353 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:43,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838883352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,354 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:43,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:43,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:43,362 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:43,364 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:43,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:43,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:43,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:43,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:43,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:43,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:43,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:43,362 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:43,362 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,368 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:43,368 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:43,368 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,368 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:43,368 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:43,369 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,369 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:43,370 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:43,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838883370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838883370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838883370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,382 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/C of c54bfce2a8106a499411906befe41bd8 into d11453e7f7c24115a38969285da124d3(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:43,382 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:43,382 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/C, priority=13, startTime=1679838823117; duration=0sec 2023-03-26 13:53:43,382 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:43,386 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/6d990461574149cd86c8c1e6b88d8501 2023-03-26 13:53:43,392 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838883390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,397 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/1618bf3e7fc144bfa7e5c408f5165d6a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1618bf3e7fc144bfa7e5c408f5165d6a 2023-03-26 13:53:43,412 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1618bf3e7fc144bfa7e5c408f5165d6a, entries=150, sequenceid=118, filesize=11.6 K 2023-03-26 13:53:43,415 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/88344d7c986741149c0da35967a9faed as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/88344d7c986741149c0da35967a9faed 2023-03-26 13:53:43,425 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/88344d7c986741149c0da35967a9faed, entries=150, sequenceid=118, filesize=11.6 K 2023-03-26 13:53:43,427 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/6d990461574149cd86c8c1e6b88d8501 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/6d990461574149cd86c8c1e6b88d8501 2023-03-26 13:53:43,439 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/6d990461574149cd86c8c1e6b88d8501, entries=150, sequenceid=118, filesize=11.6 K 2023-03-26 13:53:43,441 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~71.56 KB/73280, heapSize ~203.97 KB/208864, currentSize=107.34 KB/109920 for c54bfce2a8106a499411906befe41bd8 in 257ms, sequenceid=118, compaction requested=false 2023-03-26 13:53:43,441 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:43,470 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:43,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:43,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:43,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:43,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:43,475 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:43,475 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:43,476 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:43,476 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:43,477 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,477 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:43,480 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:43,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:43,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:43,481 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,481 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:43,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:43,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:43,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:43,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:43,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:43,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:43,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:43,486 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:43,487 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:43,488 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:43,492 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:43,492 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:43,492 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:43,492 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:43,492 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-26 13:53:43,493 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:43,493 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:43,493 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:43,493 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:43,493 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:43,493 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:43,524 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/652252ed131c43dcb06a3dd3e95a96d3 2023-03-26 13:53:43,549 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/24b3f28bdd584253b10293d4bbad287e 2023-03-26 13:53:43,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:43,559 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:43,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838883583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838883585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838883588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838883589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,596 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838883594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,605 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/ba10a5fb8a554bbcb11742d75ee3e4ba as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/ba10a5fb8a554bbcb11742d75ee3e4ba 2023-03-26 13:53:43,617 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/B of c54bfce2a8106a499411906befe41bd8 into ba10a5fb8a554bbcb11742d75ee3e4ba(size=11.8 K), total size for store is 23.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:43,619 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:43,619 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/B, priority=13, startTime=1679838823116; duration=0sec 2023-03-26 13:53:43,619 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:43,679 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-26 13:53:43,680 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-26 13:53:43,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838883689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838883691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838883693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838883696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838883893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838883894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838883898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838883898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:43,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838883899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:43,976 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/7c800c043eb34517a9eb2963bee4126c 2023-03-26 13:53:43,986 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/652252ed131c43dcb06a3dd3e95a96d3 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/652252ed131c43dcb06a3dd3e95a96d3 2023-03-26 13:53:43,995 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/652252ed131c43dcb06a3dd3e95a96d3, entries=150, sequenceid=139, filesize=11.8 K 2023-03-26 13:53:43,997 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/24b3f28bdd584253b10293d4bbad287e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/24b3f28bdd584253b10293d4bbad287e 2023-03-26 13:53:44,018 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/24b3f28bdd584253b10293d4bbad287e, entries=150, sequenceid=139, filesize=11.8 K 2023-03-26 13:53:44,019 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/7c800c043eb34517a9eb2963bee4126c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/7c800c043eb34517a9eb2963bee4126c 2023-03-26 13:53:44,032 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/7c800c043eb34517a9eb2963bee4126c, entries=150, sequenceid=139, filesize=11.8 K 2023-03-26 13:53:44,034 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for c54bfce2a8106a499411906befe41bd8 in 542ms, sequenceid=139, compaction requested=true 2023-03-26 13:53:44,035 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:44,035 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:44,035 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:44,037 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:44,037 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,037 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:44,037 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:44,041 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,041 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:44,041 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,042 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:44,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:44,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:44,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7644032c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:44,048 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:44,048 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,049 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,050 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:44,050 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:44,050 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:44,051 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,051 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:44,052 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:44,052 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:44,053 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:44,057 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:44,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:44,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:44,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:44,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:44,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:44,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:44,061 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:44,061 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:44,061 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:44,061 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:44,061 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:44,065 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,065 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:44,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,065 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:44,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:44,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:44,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,067 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:44,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:44,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:44,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:44,075 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,082 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:44,082 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:44,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:44,083 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,083 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:44,083 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:44,083 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,083 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,083 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,083 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,083 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:44,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:44,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:44,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:44,092 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:44,092 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:44,192 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:44,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:44,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:44,199 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-26 13:53:44,199 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:44,199 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:44,199 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:44,199 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:44,199 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:44,200 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:44,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:44,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:44,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:44,204 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:44,204 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:44,205 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:44,205 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:44,206 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,206 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:44,210 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:44,210 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:44,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:44,212 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,212 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:44,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:44,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:44,222 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:44,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:44,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:44,236 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:44,244 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:44,248 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:44,248 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:44,248 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:44,248 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:44,248 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:44,248 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:44,248 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:44,248 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:44,248 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:44,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:44,251 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:44,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:44,251 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:44,252 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,252 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:44,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:44,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:44,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:44,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:44,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20a432ff[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:44,253 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:44,253 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,255 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,255 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:44,255 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,255 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:44,255 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:44,255 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:44,257 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:44,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:44,257 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:44,257 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:44,257 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:44,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:44,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:44,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:44,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:44,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:44,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:44,261 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:44,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:44,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:44,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:44,261 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:44,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838884257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,263 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,263 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:44,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,263 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:44,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:44,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:44,263 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838884262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:44,269 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:44,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838884269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838884269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:44,271 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:44,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:44,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:44,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:44,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:44,277 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,277 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:44,277 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:44,277 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,277 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:44,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:44,277 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,278 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:44,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:44,278 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:44,278 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:44,278 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:44,284 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/f3aec82c6d7045b7b519992aa92ec5fb 2023-03-26 13:53:44,337 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/c1c2fa799d294cb0ab56e0146d0c4b9f 2023-03-26 13:53:44,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838884364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838884371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838884372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838884372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,378 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:44,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:44,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:44,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:44,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:44,392 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:44,392 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:44,397 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:44,397 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:44,398 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,398 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:44,399 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:44,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:44,404 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:44,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:44,406 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,406 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:44,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,406 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/5d94b4012e5e45929696615a7fb57591 2023-03-26 13:53:44,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:44,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:44,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:44,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:44,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:44,407 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:44,407 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:44,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:44,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:44,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:44,409 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:44,409 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:44,409 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:44,409 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:44,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:44,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:44,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:44,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:44,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838884406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,414 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,414 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:44,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:44,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:44,416 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,416 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:44,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:44,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:44,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ecc9f24[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:44,419 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:44,419 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,420 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/f3aec82c6d7045b7b519992aa92ec5fb as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/f3aec82c6d7045b7b519992aa92ec5fb 2023-03-26 13:53:44,422 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,422 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:44,422 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:44,422 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:44,422 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,422 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:44,425 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:44,425 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:44,425 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,425 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:44,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:44,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:44,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:44,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:44,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:44,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:44,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:44,433 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:44,433 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:44,433 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:44,433 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:44,433 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/f3aec82c6d7045b7b519992aa92ec5fb, entries=200, sequenceid=155, filesize=14.1 K 2023-03-26 13:53:44,433 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:44,437 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/c1c2fa799d294cb0ab56e0146d0c4b9f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/c1c2fa799d294cb0ab56e0146d0c4b9f 2023-03-26 13:53:44,440 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,440 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:44,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:44,440 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:44,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:44,441 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:44,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:44,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:44,443 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:44,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,449 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:44,449 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:44,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:44,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:44,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:44,449 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:44,450 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:44,450 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,449 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:44,449 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:44,450 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:44,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:44,450 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:44,453 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:44,453 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:44,453 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,455 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/c1c2fa799d294cb0ab56e0146d0c4b9f, entries=150, sequenceid=155, filesize=11.8 K 2023-03-26 13:53:44,458 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/5d94b4012e5e45929696615a7fb57591 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/5d94b4012e5e45929696615a7fb57591 2023-03-26 13:53:44,467 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/5d94b4012e5e45929696615a7fb57591, entries=150, sequenceid=155, filesize=11.8 K 2023-03-26 13:53:44,469 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for c54bfce2a8106a499411906befe41bd8 in 270ms, sequenceid=155, compaction requested=true 2023-03-26 13:53:44,469 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:44,470 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:44,470 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:53:44,470 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:53:44,470 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:44,470 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:44,473 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50440 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:53:44,473 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/A is initiating minor compaction (all files) 2023-03-26 13:53:44,473 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/A in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:44,474 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/5072ab0a565d49679f215d5f5d19ac22, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1618bf3e7fc144bfa7e5c408f5165d6a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/652252ed131c43dcb06a3dd3e95a96d3, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/f3aec82c6d7045b7b519992aa92ec5fb] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=49.3 K 2023-03-26 13:53:44,474 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48050 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:53:44,474 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 5072ab0a565d49679f215d5f5d19ac22, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1679838821849 2023-03-26 13:53:44,475 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/B is initiating minor compaction (all files) 2023-03-26 13:53:44,476 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/B in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:44,476 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/ba10a5fb8a554bbcb11742d75ee3e4ba, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/88344d7c986741149c0da35967a9faed, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/24b3f28bdd584253b10293d4bbad287e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/c1c2fa799d294cb0ab56e0146d0c4b9f] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=46.9 K 2023-03-26 13:53:44,476 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 1618bf3e7fc144bfa7e5c408f5165d6a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1679838822535 2023-03-26 13:53:44,478 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting ba10a5fb8a554bbcb11742d75ee3e4ba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1679838821849 2023-03-26 13:53:44,480 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 652252ed131c43dcb06a3dd3e95a96d3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679838823224 2023-03-26 13:53:44,480 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 88344d7c986741149c0da35967a9faed, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1679838822535 2023-03-26 13:53:44,481 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting f3aec82c6d7045b7b519992aa92ec5fb, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679838823576 2023-03-26 13:53:44,481 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 24b3f28bdd584253b10293d4bbad287e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679838823224 2023-03-26 13:53:44,482 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting c1c2fa799d294cb0ab56e0146d0c4b9f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679838823585 2023-03-26 13:53:44,495 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#A#compaction#30 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:44,496 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#B#compaction#31 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:44,553 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:44,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:44,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:44,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:44,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:44,558 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:44,558 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:44,561 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:44,561 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:44,561 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,561 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:44,564 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:44,564 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:44,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:44,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:44,564 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,565 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:44,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:44,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:44,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:44,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:44,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:44,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:44,568 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:44,568 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:44,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:44,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:44,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:44,569 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:44,569 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:44,569 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-26 13:53:44,570 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:44,570 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:44,570 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:44,570 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:44,570 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:44,570 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:44,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:44,573 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:44,597 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838884589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,599 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/0ab8531594b24d5fa50534d016be5f4f 2023-03-26 13:53:44,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838884595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838884593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,600 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838884598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838884700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838884703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838884703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838884703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,905 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838884903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838884906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838884906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:44,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838884907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:44,922 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/d500efd608ef44e0b61f0c0e8d8af035 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/d500efd608ef44e0b61f0c0e8d8af035 2023-03-26 13:53:44,922 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/f2104fee5c1b476cafec4bd059532cb2 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/f2104fee5c1b476cafec4bd059532cb2 2023-03-26 13:53:44,933 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c54bfce2a8106a499411906befe41bd8/B of c54bfce2a8106a499411906befe41bd8 into d500efd608ef44e0b61f0c0e8d8af035(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:44,933 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:44,933 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/B, priority=12, startTime=1679838824470; duration=0sec 2023-03-26 13:53:44,933 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:44,933 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:53:44,936 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c54bfce2a8106a499411906befe41bd8/A of c54bfce2a8106a499411906befe41bd8 into f2104fee5c1b476cafec4bd059532cb2(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:44,936 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:44,936 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/A, priority=12, startTime=1679838824470; duration=0sec 2023-03-26 13:53:44,936 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:44,938 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48050 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:53:44,938 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/C is initiating minor compaction (all files) 2023-03-26 13:53:44,938 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/C in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:44,938 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/d11453e7f7c24115a38969285da124d3, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/6d990461574149cd86c8c1e6b88d8501, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/7c800c043eb34517a9eb2963bee4126c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/5d94b4012e5e45929696615a7fb57591] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=46.9 K 2023-03-26 13:53:44,939 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting d11453e7f7c24115a38969285da124d3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=95, earliestPutTs=1679838821849 2023-03-26 13:53:44,939 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 6d990461574149cd86c8c1e6b88d8501, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1679838822535 2023-03-26 13:53:44,940 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 7c800c043eb34517a9eb2963bee4126c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679838823224 2023-03-26 13:53:44,941 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 5d94b4012e5e45929696615a7fb57591, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679838823585 2023-03-26 13:53:44,954 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#C#compaction#34 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:44,977 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/7b560923a364478ab71d625dc5756b43 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/7b560923a364478ab71d625dc5756b43 2023-03-26 13:53:44,985 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c54bfce2a8106a499411906befe41bd8/C of c54bfce2a8106a499411906befe41bd8 into 7b560923a364478ab71d625dc5756b43(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:44,985 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:44,985 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/C, priority=12, startTime=1679838824470; duration=0sec 2023-03-26 13:53:44,986 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:45,027 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/74042ed7fef547b1a408779e1e219845 2023-03-26 13:53:45,053 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=172 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/a9dd4396d2c541eeab3d0cdbe3cf8a49 2023-03-26 13:53:45,062 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/0ab8531594b24d5fa50534d016be5f4f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/0ab8531594b24d5fa50534d016be5f4f 2023-03-26 13:53:45,071 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/0ab8531594b24d5fa50534d016be5f4f, entries=150, sequenceid=172, filesize=11.8 K 2023-03-26 13:53:45,073 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/74042ed7fef547b1a408779e1e219845 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/74042ed7fef547b1a408779e1e219845 2023-03-26 13:53:45,081 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/74042ed7fef547b1a408779e1e219845, entries=150, sequenceid=172, filesize=11.8 K 2023-03-26 13:53:45,082 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/a9dd4396d2c541eeab3d0cdbe3cf8a49 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/a9dd4396d2c541eeab3d0cdbe3cf8a49 2023-03-26 13:53:45,094 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/a9dd4396d2c541eeab3d0cdbe3cf8a49, entries=150, sequenceid=172, filesize=11.8 K 2023-03-26 13:53:45,095 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for c54bfce2a8106a499411906befe41bd8 in 526ms, sequenceid=172, compaction requested=false 2023-03-26 13:53:45,096 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:45,096 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:45,096 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:45,097 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:45,097 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,097 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:45,097 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:45,100 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,100 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:45,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:45,101 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,101 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:45,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:45,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:45,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:45,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:45,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c17a012[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:45,103 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:45,103 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,104 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,105 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:45,105 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:45,105 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:45,105 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,105 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:45,106 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:45,107 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:45,107 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,107 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:45,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:45,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:45,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:45,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:45,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:45,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:45,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:45,111 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:45,111 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:45,111 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:45,111 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:45,111 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:45,113 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,113 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:45,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,113 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:45,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:45,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:45,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:45,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:45,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,114 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:45,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:45,118 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,122 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,122 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,122 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,122 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,122 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,122 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:45,122 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:45,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:45,122 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,123 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:45,123 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:45,123 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:45,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:45,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:45,123 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:45,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:45,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:45,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:45,213 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-26 13:53:45,215 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:45,215 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:45,215 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:45,215 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:45,215 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:45,215 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:45,224 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:45,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:45,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:45,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:45,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:45,230 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:45,230 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:45,233 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:45,233 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:45,233 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,233 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,235 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:45,235 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:45,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:45,235 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,235 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:45,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:45,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:45,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:45,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:45,237 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:45,237 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:45,239 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:45,239 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:45,239 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:45,239 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:45,240 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:45,240 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:45,240 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:45,240 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:45,240 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:45,240 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,240 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:45,240 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:45,241 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,241 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:45,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:45,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:45,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:45,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:45,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:45,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:45,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a1c81df[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:45,243 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:45,243 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,245 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,245 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,245 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:45,245 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:45,245 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:45,245 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:45,247 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,247 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:45,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:45,248 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:45,248 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:45,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:45,251 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:45,251 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:45,251 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:45,251 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:45,251 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:45,251 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:45,252 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,252 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,252 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:45,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:45,253 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:45,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838885269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,273 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:45,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838885270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,274 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:45,274 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:45,274 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:45,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:45,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:45,274 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:45,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:45,278 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:45,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838885271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:45,274 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,278 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:45,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:45,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838885272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:45,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838885374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,379 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:45,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:45,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:45,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:45,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838885380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:45,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:45,384 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:45,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:45,384 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:45,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838885380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,384 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:45,384 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:45,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:45,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838885381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,386 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,386 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,389 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:45,389 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:45,389 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,389 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:45,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:45,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:45,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:45,390 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:45,391 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:45,391 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:45,391 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:45,391 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:45,391 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:45,391 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:45,391 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:45,391 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:45,391 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:45,391 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:45,391 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,391 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:45,391 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:45,393 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:45,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:45,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b2e6da9[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:45,395 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:45,395 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,396 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:45,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:45,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:45,397 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,397 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:45,398 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:45,399 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:45,399 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:45,399 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:45,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:45,402 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:45,402 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:45,402 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:45,402 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:45,402 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:45,404 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:45,404 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,405 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:45,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:45,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:45,407 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:45,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,408 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:45,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:45,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838885412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,416 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:45,416 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:45,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:45,416 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:45,416 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:45,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:45,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:45,416 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,417 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,417 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,417 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,417 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,417 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,417 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:45,417 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:45,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:45,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:45,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:45,517 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:45,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:45,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:45,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:45,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:45,522 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:45,522 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:45,522 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:45,522 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:45,522 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,523 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,525 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,525 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:45,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:45,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:45,525 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,525 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:45,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:45,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:45,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:45,526 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:45,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:45,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:45,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:45,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:45,527 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:45,527 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:45,527 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:45,527 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:45,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:45,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:45,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:45,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:45,529 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,529 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:45,529 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,530 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:45,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:45,531 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:45,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41f1f257[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:45,531 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,532 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,532 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:45,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:45,532 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,533 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:45,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:45,534 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:45,534 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,534 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:45,534 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:45,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:45,537 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:45,537 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:45,537 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:45,537 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:45,537 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:45,541 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,541 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,541 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:45,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:45,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,543 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,544 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,547 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,548 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,548 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,548 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,548 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,548 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,548 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:45,548 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,548 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:45,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:45,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:45,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:45,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:45,548 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:45,548 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:45,548 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:45,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:45,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:45,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:45,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838885577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:45,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838885585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:45,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838885586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:45,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838885587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,636 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/e556cfe48d0940ccaca5e6733ac0b563 2023-03-26 13:53:45,648 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:45,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:45,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:45,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:45,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:45,656 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:45,656 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:45,656 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:45,656 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:45,657 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,657 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,658 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,658 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:45,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:45,658 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,658 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:45,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:45,664 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:45,664 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:45,664 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:45,664 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:45,664 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/26cb4f96448b40dd938697d478a3449c 2023-03-26 13:53:45,664 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:45,664 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:45,664 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:45,664 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:45,665 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:45,665 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:45,667 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:45,667 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:45,667 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,667 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:45,667 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:45,670 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:45,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:45,670 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,671 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:45,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:45,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:45,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a7dd4b[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:45,674 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:45,674 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,676 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,677 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:45,677 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:45,677 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:45,677 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,678 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:45,679 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:45,679 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:45,679 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,679 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:45,683 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:45,684 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:45,684 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:45,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:45,684 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:45,684 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:45,685 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,685 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,685 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:45,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:45,687 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,687 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=194 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/ffba6244a7694d88a6eeb82362405844 2023-03-26 13:53:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:45,688 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:45,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:45,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:45,692 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:45,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:45,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:45,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:45,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:45,692 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:45,693 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:45,694 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,694 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,694 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,694 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,694 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,694 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,694 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,695 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/e556cfe48d0940ccaca5e6733ac0b563 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e556cfe48d0940ccaca5e6733ac0b563 2023-03-26 13:53:45,710 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e556cfe48d0940ccaca5e6733ac0b563, entries=200, sequenceid=194, filesize=14.1 K 2023-03-26 13:53:45,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/26cb4f96448b40dd938697d478a3449c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/26cb4f96448b40dd938697d478a3449c 2023-03-26 13:53:45,720 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/26cb4f96448b40dd938697d478a3449c, entries=150, sequenceid=194, filesize=11.8 K 2023-03-26 13:53:45,721 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/ffba6244a7694d88a6eeb82362405844 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/ffba6244a7694d88a6eeb82362405844 2023-03-26 13:53:45,726 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/ffba6244a7694d88a6eeb82362405844, entries=150, sequenceid=194, filesize=11.8 K 2023-03-26 13:53:45,727 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for c54bfce2a8106a499411906befe41bd8 in 514ms, sequenceid=194, compaction requested=true 2023-03-26 13:53:45,728 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:45,728 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:45,728 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:45,728 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:45,728 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:45,731 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:45,732 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:45,732 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/B is initiating minor compaction (all files) 2023-03-26 13:53:45,732 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/B in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:45,732 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38840 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:45,732 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/d500efd608ef44e0b61f0c0e8d8af035, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/74042ed7fef547b1a408779e1e219845, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/26cb4f96448b40dd938697d478a3449c] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=35.6 K 2023-03-26 13:53:45,732 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/A is initiating minor compaction (all files) 2023-03-26 13:53:45,732 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/A in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:45,733 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/f2104fee5c1b476cafec4bd059532cb2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/0ab8531594b24d5fa50534d016be5f4f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e556cfe48d0940ccaca5e6733ac0b563] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=37.9 K 2023-03-26 13:53:45,733 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting d500efd608ef44e0b61f0c0e8d8af035, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679838823585 2023-03-26 13:53:45,734 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting f2104fee5c1b476cafec4bd059532cb2, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679838823585 2023-03-26 13:53:45,735 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 74042ed7fef547b1a408779e1e219845, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1679838824234 2023-03-26 13:53:45,735 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 0ab8531594b24d5fa50534d016be5f4f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1679838824234 2023-03-26 13:53:45,736 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 26cb4f96448b40dd938697d478a3449c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1679838824593 2023-03-26 13:53:45,736 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting e556cfe48d0940ccaca5e6733ac0b563, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1679838824593 2023-03-26 13:53:45,747 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#A#compaction#39 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:45,747 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#B#compaction#40 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:45,793 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:45,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:45,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:45,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:45,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:45,797 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:45,797 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:45,797 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:45,797 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:45,798 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,798 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,800 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:45,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:45,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:45,801 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,801 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:45,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:45,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:45,802 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:45,802 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:45,802 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:45,802 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:45,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:45,803 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:45,803 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:45,803 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-26 13:53:45,803 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:45,803 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:45,803 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:45,803 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:45,803 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:45,804 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:45,815 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/e41d4c0a3a7b4cfea47b72c9d01a66a9 2023-03-26 13:53:45,846 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/10f83c24bb174200bb34d1e447e7271b 2023-03-26 13:53:45,871 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/3a75616ca9344f82946d81d412c8a980 2023-03-26 13:53:45,879 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/e41d4c0a3a7b4cfea47b72c9d01a66a9 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e41d4c0a3a7b4cfea47b72c9d01a66a9 2023-03-26 13:53:45,886 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e41d4c0a3a7b4cfea47b72c9d01a66a9, entries=150, sequenceid=207, filesize=11.8 K 2023-03-26 13:53:45,887 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/10f83c24bb174200bb34d1e447e7271b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/10f83c24bb174200bb34d1e447e7271b 2023-03-26 13:53:45,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:45,893 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:45,902 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/10f83c24bb174200bb34d1e447e7271b, entries=150, sequenceid=207, filesize=11.8 K 2023-03-26 13:53:45,903 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/3a75616ca9344f82946d81d412c8a980 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/3a75616ca9344f82946d81d412c8a980 2023-03-26 13:53:45,909 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/3a75616ca9344f82946d81d412c8a980, entries=150, sequenceid=207, filesize=11.8 K 2023-03-26 13:53:45,911 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=46.96 KB/48090 for c54bfce2a8106a499411906befe41bd8 in 108ms, sequenceid=207, compaction requested=true 2023-03-26 13:53:45,911 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:45,911 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:45,911 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:45,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:45,912 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:45,912 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 13:53:45,912 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,912 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:45,912 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:45,912 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:45,913 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:45,913 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:45,913 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:45,913 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:45,913 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:45,914 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,915 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:45,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:45,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:45,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:45,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4efbb6f9[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:45,919 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:45,919 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,921 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,921 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:45,921 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:45,921 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:45,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:45,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:45,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:45,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,925 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:45,925 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:45,925 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:45,925 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:45,925 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:45,925 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:45,925 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:45,927 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,928 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:45,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,928 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:45,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:45,928 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:45,931 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:45,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:45,933 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/470f2b5f49124218adb567b4141d3311 2023-03-26 13:53:45,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:45,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:45,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:45,939 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:45,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:45,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:45,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:45,939 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:45,939 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:45,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:45,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:45,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:45,940 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,940 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,940 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:45,940 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:45,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:45,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:45,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:45,952 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/7e63c6c93b9e4de6b23b032de3df64d5 2023-03-26 13:53:45,954 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:45,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838885951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:45,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838825941, max timestamp=1679838825951Num uniques 859; 2023-03-26 13:53:45,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:53:45,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838825941, max timestamp=1679838825951Num uniques 859; 2023-03-26 13:53:45,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:53:45,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838825941, max timestamp=1679838825951Num uniques 859; 2023-03-26 13:53:45,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:53:45,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:45,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838885987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:46,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838885994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:46,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838885999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,039 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:46,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:46,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:46,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:46,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:46,044 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:46,044 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:46,044 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:46,044 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:46,044 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,044 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:46,047 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:46,047 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:46,048 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:46,048 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:46,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:46,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:46,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:46,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:46,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:46,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:46,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:46,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:46,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:46,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:46,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:46,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:46,055 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,055 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:46,055 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:46,056 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,056 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:46,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:46,057 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:46,057 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:46,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:46,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:46,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:46,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:46,058 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:46,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:46,058 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838886056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1660d9aa[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:46,061 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,061 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:46,061 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:46,061 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:46,061 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,061 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:46,063 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,063 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:46,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,063 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:46,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:46,063 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:46,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:46,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:46,067 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:46,067 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:46,067 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:46,067 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:46,067 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:46,071 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,071 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,071 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:46,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:46,071 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:46,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:46,073 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,077 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:46,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:46,077 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:46,077 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,077 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:46,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:46,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:46,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:46,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:46,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:46,078 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:46,078 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:46,078 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:46,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,080 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,080 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:46,099 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:46,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838886099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:46,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838886102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:46,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838886103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,175 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/e21393b7af614635a6b925c1214c9a9c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e21393b7af614635a6b925c1214c9a9c 2023-03-26 13:53:46,178 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:46,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:46,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:46,187 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/282f8aaad2434ae78de29ad00f0a2865 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/282f8aaad2434ae78de29ad00f0a2865 2023-03-26 13:53:46,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:46,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:46,192 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:46,192 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:46,192 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:46,192 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:46,193 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,193 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:46,194 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/A of c54bfce2a8106a499411906befe41bd8 into e21393b7af614635a6b925c1214c9a9c(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:46,194 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:46,194 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/A, priority=13, startTime=1679838825728; duration=0sec 2023-03-26 13:53:46,194 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:46,194 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:53:46,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:46,195 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:46,195 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,195 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:46,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:46,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:46,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:46,197 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:53:46,197 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/C is initiating minor compaction (all files) 2023-03-26 13:53:46,197 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/C in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:46,198 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/7b560923a364478ab71d625dc5756b43, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/a9dd4396d2c541eeab3d0cdbe3cf8a49, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/ffba6244a7694d88a6eeb82362405844, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/3a75616ca9344f82946d81d412c8a980] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=47.3 K 2023-03-26 13:53:46,198 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 7b560923a364478ab71d625dc5756b43, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1679838823585 2023-03-26 13:53:46,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:46,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:46,212 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:46,212 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:46,212 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting a9dd4396d2c541eeab3d0cdbe3cf8a49, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=172, earliestPutTs=1679838824234 2023-03-26 13:53:46,212 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:46,212 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:46,212 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:46,212 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:46,212 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:46,212 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:46,213 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:46,213 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:46,213 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:46,213 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,213 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting ffba6244a7694d88a6eeb82362405844, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1679838824593 2023-03-26 13:53:46,213 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:46,217 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:46,217 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 3a75616ca9344f82946d81d412c8a980, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1679838825268 2023-03-26 13:53:46,218 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/B of c54bfce2a8106a499411906befe41bd8 into 282f8aaad2434ae78de29ad00f0a2865(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:46,218 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:46,218 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/B, priority=13, startTime=1679838825728; duration=0sec 2023-03-26 13:53:46,218 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:46,224 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,224 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:46,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:46,225 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,225 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:46,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:46,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:46,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:46,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58cde27b[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:46,228 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:46,228 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,229 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,229 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:46,229 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:46,230 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:46,230 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,230 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:46,231 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,231 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:46,231 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:46,231 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:46,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:46,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:46,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:46,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:46,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:46,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:46,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:46,242 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:46,242 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:46,242 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:46,242 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:46,242 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:46,242 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#C#compaction#47 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:46,244 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,244 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:46,244 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:46,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:46,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:46,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:46,246 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:46,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,248 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:46,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,251 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:46,252 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:46,252 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,252 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:46,252 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:46,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:46,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:46,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:46,252 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,252 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,252 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,252 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,252 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:46,252 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:46,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:46,252 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:46,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:46,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:46,262 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:46,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838886261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,265 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/4f6765dd3c8641fd90a27d8e6526cd20 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/4f6765dd3c8641fd90a27d8e6526cd20 2023-03-26 13:53:46,273 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c54bfce2a8106a499411906befe41bd8/C of c54bfce2a8106a499411906befe41bd8 into 4f6765dd3c8641fd90a27d8e6526cd20(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:46,273 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:46,273 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/C, priority=12, startTime=1679838825731; duration=0sec 2023-03-26 13:53:46,273 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:46,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:46,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838886301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:46,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838886307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:46,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838886308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,353 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:46,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:46,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:46,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:46,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:46,357 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:46,357 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:46,357 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:46,357 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:46,358 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,358 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:46,359 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,359 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:46,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:46,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:46,360 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,360 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:46,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:46,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:46,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:46,361 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:46,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:46,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:46,361 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:46,361 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:46,361 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:46,361 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:46,361 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:46,361 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:46,362 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:46,362 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:46,362 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:46,362 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:46,364 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,364 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:46,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:46,365 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,365 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:46,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:46,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:46,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66be21ac[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:46,366 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:46,367 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,368 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,368 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:46,368 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:46,368 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,368 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:46,368 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:46,371 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:46,371 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:46,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:46,371 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:46,372 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:46,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:46,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:46,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:46,374 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:46,374 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:46,374 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:46,374 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:46,374 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:46,376 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,376 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,376 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:46,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:46,377 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:46,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:46,378 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:46,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:46,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,382 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:46,382 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:46,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:46,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:46,382 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,382 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:46,382 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:46,382 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:46,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:46,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:46,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:46,382 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:46,382 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:46,382 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,382 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,382 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:46,382 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:46,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:46,384 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/f00d83c381c84a59be8c978dcd86a0ab 2023-03-26 13:53:46,391 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/470f2b5f49124218adb567b4141d3311 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/470f2b5f49124218adb567b4141d3311 2023-03-26 13:53:46,397 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/470f2b5f49124218adb567b4141d3311, entries=250, sequenceid=219, filesize=16.4 K 2023-03-26 13:53:46,398 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/7e63c6c93b9e4de6b23b032de3df64d5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/7e63c6c93b9e4de6b23b032de3df64d5 2023-03-26 13:53:46,403 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/7e63c6c93b9e4de6b23b032de3df64d5, entries=150, sequenceid=219, filesize=11.8 K 2023-03-26 13:53:46,404 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/f00d83c381c84a59be8c978dcd86a0ab as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/f00d83c381c84a59be8c978dcd86a0ab 2023-03-26 13:53:46,410 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/f00d83c381c84a59be8c978dcd86a0ab, entries=150, sequenceid=219, filesize=11.8 K 2023-03-26 13:53:46,412 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for c54bfce2a8106a499411906befe41bd8 in 500ms, sequenceid=219, compaction requested=true 2023-03-26 13:53:46,412 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:46,412 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:46,412 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:46,412 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:46,412 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:46,412 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:46,413 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41332 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:46,413 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:46,413 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/A is initiating minor compaction (all files) 2023-03-26 13:53:46,414 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/B is initiating minor compaction (all files) 2023-03-26 13:53:46,414 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/A in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:46,414 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/B in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:46,414 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e21393b7af614635a6b925c1214c9a9c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e41d4c0a3a7b4cfea47b72c9d01a66a9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/470f2b5f49124218adb567b4141d3311] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=40.4 K 2023-03-26 13:53:46,414 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/282f8aaad2434ae78de29ad00f0a2865, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/10f83c24bb174200bb34d1e447e7271b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/7e63c6c93b9e4de6b23b032de3df64d5] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=35.7 K 2023-03-26 13:53:46,414 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting e21393b7af614635a6b925c1214c9a9c, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1679838824593 2023-03-26 13:53:46,415 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 282f8aaad2434ae78de29ad00f0a2865, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=194, earliestPutTs=1679838824593 2023-03-26 13:53:46,417 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 10f83c24bb174200bb34d1e447e7271b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1679838825268 2023-03-26 13:53:46,417 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting e41d4c0a3a7b4cfea47b72c9d01a66a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1679838825268 2023-03-26 13:53:46,417 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 7e63c6c93b9e4de6b23b032de3df64d5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679838825899 2023-03-26 13:53:46,417 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 470f2b5f49124218adb567b4141d3311, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679838825897 2023-03-26 13:53:46,433 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#A#compaction#49 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:46,433 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#B#compaction#48 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:46,451 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/e1cad71b100e4832b4a61e2e47fb1622 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e1cad71b100e4832b4a61e2e47fb1622 2023-03-26 13:53:46,457 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/A of c54bfce2a8106a499411906befe41bd8 into e1cad71b100e4832b4a61e2e47fb1622(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:46,457 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:46,457 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/A, priority=13, startTime=1679838826412; duration=0sec 2023-03-26 13:53:46,457 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:46,457 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-26 13:53:46,458 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 13:53:46,458 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 13:53:46,458 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. because compaction request was cancelled 2023-03-26 13:53:46,482 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:46,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:46,485 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:46,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:46,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:46,487 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:46,487 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:46,487 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:46,487 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:46,487 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,487 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:46,489 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:46,489 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:46,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:46,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:46,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:46,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:46,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:46,491 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:46,492 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,492 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:46,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:46,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:46,493 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:46,493 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:46,493 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:46,494 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:46,494 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:46,494 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:46,494 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:46,494 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-26 13:53:46,494 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:46,494 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:46,494 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:46,495 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:46,495 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:46,495 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:46,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:46,567 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:46,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:46,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838886587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:46,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838886606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:46,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838886612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:46,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838886613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:46,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838886692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,854 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/789e0c6d48a44ac1a96d4cd784646bc1 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/789e0c6d48a44ac1a96d4cd784646bc1 2023-03-26 13:53:46,864 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/B of c54bfce2a8106a499411906befe41bd8 into 789e0c6d48a44ac1a96d4cd784646bc1(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:46,864 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:46,864 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/B, priority=13, startTime=1679838826412; duration=0sec 2023-03-26 13:53:46,864 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:46,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:46,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838886897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:46,905 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/026b7695d64246139329ef45d1e960a9 2023-03-26 13:53:47,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:47,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838887115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:47,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838887117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:47,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838887118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:47,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838887200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,339 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/cd207e8f7d5744e7b394a3ac6bfef917 2023-03-26 13:53:47,359 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/e638a5c110284999a0b0206358a2b383 2023-03-26 13:53:47,369 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/026b7695d64246139329ef45d1e960a9 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/026b7695d64246139329ef45d1e960a9 2023-03-26 13:53:47,377 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/026b7695d64246139329ef45d1e960a9, entries=150, sequenceid=259, filesize=11.8 K 2023-03-26 13:53:47,381 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/cd207e8f7d5744e7b394a3ac6bfef917 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/cd207e8f7d5744e7b394a3ac6bfef917 2023-03-26 13:53:47,388 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/cd207e8f7d5744e7b394a3ac6bfef917, entries=150, sequenceid=259, filesize=11.8 K 2023-03-26 13:53:47,389 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/e638a5c110284999a0b0206358a2b383 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e638a5c110284999a0b0206358a2b383 2023-03-26 13:53:47,409 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e638a5c110284999a0b0206358a2b383, entries=150, sequenceid=259, filesize=11.8 K 2023-03-26 13:53:47,411 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for c54bfce2a8106a499411906befe41bd8 in 917ms, sequenceid=259, compaction requested=true 2023-03-26 13:53:47,411 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:47,411 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:47,411 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:47,413 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:47,413 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,413 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:47,413 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:47,418 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,419 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:47,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:47,419 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,419 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:47,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:47,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:47,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:47,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:47,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75611ffc[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:47,422 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:47,422 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,424 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:47,424 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:47,425 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,425 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:47,426 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:47,426 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:47,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:47,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:47,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:47,432 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 13:53:47,432 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:47,433 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:47,433 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:47,433 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:47,433 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:47,433 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:47,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:47,436 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:47,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:47,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:47,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:47,436 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:47,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,442 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:47,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:47,455 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:47,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:47,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,457 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:47,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,462 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,467 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,467 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:47,467 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:47,467 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:47,467 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:47,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:47,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:47,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:47,467 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:47,467 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,467 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,467 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,467 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:47,468 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:47,468 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:47,493 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/a59e2b751b60431684f290cd90d11412 2023-03-26 13:53:47,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:47,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838887525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,568 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:47,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:47,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:47,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:47,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:47,584 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:47,584 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:47,584 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:47,584 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:47,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:47,586 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:47,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:47,587 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,587 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:47,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:47,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:47,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:47,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:47,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:47,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:47,609 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:47,609 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:47,609 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:47,622 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:47,622 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:47,622 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:47,622 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:47,622 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:47,622 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:47,622 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:47,622 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:47,622 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,622 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:47,622 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:47,625 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:47,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:47,625 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,626 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,626 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:47,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:47,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:47,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:47,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:47,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@655c2573[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:47,629 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:47,629 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:47,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838887630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,632 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,632 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:47,632 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:47,632 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:47,632 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,632 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:47,634 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:47,634 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:47,634 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:47,635 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:47,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:47,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:47,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:47,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:47,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:47,639 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:47,639 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:47,639 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:47,639 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:47,639 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:47,641 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,641 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:47,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:47,642 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:47,643 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,652 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,656 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:47,656 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:47,656 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,656 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:47,656 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:47,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:47,657 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:47,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:47,657 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,657 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:47,657 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:47,657 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:47,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:47,657 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,657 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:47,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838887709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,757 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:47,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:47,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:47,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:47,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:47,765 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:47,765 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:47,765 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:47,765 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:47,765 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,766 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:47,768 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,768 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:47,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:47,769 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,769 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:47,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:47,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:47,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:47,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:47,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:47,772 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:47,772 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:47,772 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:47,781 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:47,781 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:47,781 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:47,781 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:47,781 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:47,781 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:47,781 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:47,781 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:47,781 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,781 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:47,781 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:47,783 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,783 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:47,784 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,784 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:47,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:47,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:47,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52b5cd99[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:47,790 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:47,790 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,791 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,792 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:47,792 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:47,792 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:47,793 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,793 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:47,794 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:47,794 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,794 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:47,794 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:47,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:47,796 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:47,797 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:47,797 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:47,797 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:47,797 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:47,801 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,801 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,801 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:47,805 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:47,806 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:47,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,810 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,810 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:47,810 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:47,810 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,810 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,810 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,810 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,810 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:47,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:47,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:47,811 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:47,811 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:47,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:47,821 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:47,821 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:47,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:47,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:47,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:47,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:47,838 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:47,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838887835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,911 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:47,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:47,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:47,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:47,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:47,919 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:47,919 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:47,919 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:47,919 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:47,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:47,926 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:47,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:47,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:47,926 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:47,927 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,927 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:47,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:47,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:47,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:47,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:47,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:47,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:47,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:47,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:47,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:47,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:47,929 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:47,929 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:47,929 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:47,929 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:47,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:47,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:47,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:47,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:47,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,931 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:47,932 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,932 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:47,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:47,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:47,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:47,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:47,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45913522[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:47,933 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:47,934 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,937 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/91d4e3c86c2042838c22d2a9dda90621 2023-03-26 13:53:47,938 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,938 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:47,938 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:47,938 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:47,939 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,939 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:47,940 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:47,940 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,940 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:47,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:47,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:47,940 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:47,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:47,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:47,944 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:47,944 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:47,944 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:47,944 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:47,944 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:47,946 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,946 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,946 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:47,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:47,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:47,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,953 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:47,954 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/cda455e37f044645a987a211e2ff3416 2023-03-26 13:53:47,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:47,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,956 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,967 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:47,967 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,967 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:47,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:47,967 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/a59e2b751b60431684f290cd90d11412 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/a59e2b751b60431684f290cd90d11412 2023-03-26 13:53:47,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:47,967 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:47,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:47,967 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:47,968 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:47,968 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,968 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:47,968 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:47,967 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:47,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:47,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:47,968 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:47,969 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:47,975 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/a59e2b751b60431684f290cd90d11412, entries=200, sequenceid=272, filesize=14.3 K 2023-03-26 13:53:47,976 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/91d4e3c86c2042838c22d2a9dda90621 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/91d4e3c86c2042838c22d2a9dda90621 2023-03-26 13:53:47,982 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/91d4e3c86c2042838c22d2a9dda90621, entries=150, sequenceid=272, filesize=11.9 K 2023-03-26 13:53:47,984 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/cda455e37f044645a987a211e2ff3416 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/cda455e37f044645a987a211e2ff3416 2023-03-26 13:53:47,998 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/cda455e37f044645a987a211e2ff3416, entries=150, sequenceid=272, filesize=11.9 K 2023-03-26 13:53:47,999 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c54bfce2a8106a499411906befe41bd8 in 567ms, sequenceid=272, compaction requested=true 2023-03-26 13:53:47,999 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:47,999 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:47,999 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:47,999 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:47,999 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:48,000 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:48,001 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39244 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:48,001 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/A is initiating minor compaction (all files) 2023-03-26 13:53:48,001 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/A in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:48,001 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e1cad71b100e4832b4a61e2e47fb1622, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/026b7695d64246139329ef45d1e960a9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/a59e2b751b60431684f290cd90d11412] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=38.3 K 2023-03-26 13:53:48,002 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36804 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:48,002 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/B is initiating minor compaction (all files) 2023-03-26 13:53:48,003 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/B in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:48,003 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/789e0c6d48a44ac1a96d4cd784646bc1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/cd207e8f7d5744e7b394a3ac6bfef917, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/91d4e3c86c2042838c22d2a9dda90621] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=35.9 K 2023-03-26 13:53:48,003 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting e1cad71b100e4832b4a61e2e47fb1622, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679838825899 2023-03-26 13:53:48,006 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 789e0c6d48a44ac1a96d4cd784646bc1, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679838825899 2023-03-26 13:53:48,007 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 026b7695d64246139329ef45d1e960a9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679838825969 2023-03-26 13:53:48,008 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting cd207e8f7d5744e7b394a3ac6bfef917, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679838825969 2023-03-26 13:53:48,008 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting a59e2b751b60431684f290cd90d11412, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1679838826578 2023-03-26 13:53:48,008 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 91d4e3c86c2042838c22d2a9dda90621, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1679838826578 2023-03-26 13:53:48,020 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#B#compaction#56 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:48,026 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#A#compaction#57 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:48,068 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:48,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:48,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:48,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:48,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:48,073 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:48,073 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:48,073 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:48,074 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:48,074 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,074 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,075 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:48,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:48,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:48,076 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,079 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,079 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:48,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:48,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:48,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:48,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:48,080 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:48,080 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:48,080 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:48,081 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:48,081 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:48,081 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:48,081 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:48,081 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-26 13:53:48,081 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:48,082 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:48,082 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:48,082 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:48,082 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:48,082 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:48,106 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/bf6497e677014c8b8c880b3c4e6d018e 2023-03-26 13:53:48,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:48,124 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:48,133 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/a05dc142bab644168c737351019ae34c 2023-03-26 13:53:48,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838888136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838888138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838888140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39082 deadline: 1679838888141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,158 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/8e30cc5b54d0441a95b00eeb56924aa5 2023-03-26 13:53:48,166 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/bf6497e677014c8b8c880b3c4e6d018e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/bf6497e677014c8b8c880b3c4e6d018e 2023-03-26 13:53:48,171 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/bf6497e677014c8b8c880b3c4e6d018e, entries=150, sequenceid=292, filesize=11.9 K 2023-03-26 13:53:48,172 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/a05dc142bab644168c737351019ae34c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/a05dc142bab644168c737351019ae34c 2023-03-26 13:53:48,179 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/a05dc142bab644168c737351019ae34c, entries=150, sequenceid=292, filesize=11.9 K 2023-03-26 13:53:48,180 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/8e30cc5b54d0441a95b00eeb56924aa5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/8e30cc5b54d0441a95b00eeb56924aa5 2023-03-26 13:53:48,186 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/8e30cc5b54d0441a95b00eeb56924aa5, entries=150, sequenceid=292, filesize=11.9 K 2023-03-26 13:53:48,187 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for c54bfce2a8106a499411906befe41bd8 in 106ms, sequenceid=292, compaction requested=true 2023-03-26 13:53:48,187 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:48,187 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:48,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:48,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:48,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:48,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:48,189 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,189 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:48,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:48,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:48,190 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,190 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:48,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:48,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:48,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:48,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38f24c93[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:48,194 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:48,194 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,196 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:48,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:48,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:48,197 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,197 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:48,199 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,199 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:48,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:48,199 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:48,200 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:48,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:48,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:48,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:48,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:48,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:48,203 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:48,203 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:48,203 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:48,203 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:48,203 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:48,205 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,205 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:48,205 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:48,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:48,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,207 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:48,208 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:48,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,212 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,212 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:48,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:48,213 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:48,212 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:48,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:48,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:48,213 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:48,213 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:48,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:48,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:48,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:48,244 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-26 13:53:48,244 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:48,244 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:48,245 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:48,245 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:48,245 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:48,245 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:48,267 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/275d709fb2b446ec9d14eee31063ae44 2023-03-26 13:53:48,296 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/e15126c2469f4a72ac54484689209501 2023-03-26 13:53:48,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838888296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838888296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39074 deadline: 1679838888298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,313 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:48,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:48,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:48,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:48,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:48,333 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:48,333 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:48,333 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:48,333 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:48,333 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,333 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,335 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:48,335 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:48,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,336 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,336 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:48,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:48,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:48,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:48,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:48,350 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:48,350 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:48,350 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/b2dbdb9460fa43c7865c00cc5e7eb885 2023-03-26 13:53:48,350 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:48,351 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:48,351 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:48,351 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:48,351 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:48,351 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:48,351 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:48,351 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:48,351 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:48,351 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,351 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:48,351 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:48,356 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,356 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:48,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:48,357 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,357 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:48,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,359 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/275d709fb2b446ec9d14eee31063ae44 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/275d709fb2b446ec9d14eee31063ae44 2023-03-26 13:53:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:48,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:48,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c018143[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:48,360 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:48,361 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,364 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,364 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:48,364 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:48,364 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:48,364 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,364 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:48,367 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:48,367 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:48,367 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:48,367 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:48,368 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/275d709fb2b446ec9d14eee31063ae44, entries=150, sequenceid=305, filesize=11.9 K 2023-03-26 13:53:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:48,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,371 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/e15126c2469f4a72ac54484689209501 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e15126c2469f4a72ac54484689209501 2023-03-26 13:53:48,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:48,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:48,375 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:48,375 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:48,375 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:48,375 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:48,375 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:48,381 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,381 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:48,381 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:48,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:48,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:48,383 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,385 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,388 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e15126c2469f4a72ac54484689209501, entries=150, sequenceid=305, filesize=11.9 K 2023-03-26 13:53:48,390 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/b2dbdb9460fa43c7865c00cc5e7eb885 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/b2dbdb9460fa43c7865c00cc5e7eb885 2023-03-26 13:53:48,391 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:48,391 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:48,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:48,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:48,391 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,391 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,391 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:48,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:48,391 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,391 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,391 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,391 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:48,391 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:48,391 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:48,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:48,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:48,396 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/b2dbdb9460fa43c7865c00cc5e7eb885, entries=150, sequenceid=305, filesize=11.9 K 2023-03-26 13:53:48,397 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for c54bfce2a8106a499411906befe41bd8 in 153ms, sequenceid=305, compaction requested=true 2023-03-26 13:53:48,397 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:48,398 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-26 13:53:48,398 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-26 13:53:48,398 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-26 13:53:48,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:48,409 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-26 13:53:48,409 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:48,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:48,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:48,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:48,410 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:48,410 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:48,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838888429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838888430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838888436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,442 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/433589000cc34e38831fa0c212a50de1 2023-03-26 13:53:48,472 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/87dbc2256ca04fcb83683dc95bd69c4e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/87dbc2256ca04fcb83683dc95bd69c4e 2023-03-26 13:53:48,473 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/873b959ad55f40858dc949cb6d812bde as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/873b959ad55f40858dc949cb6d812bde 2023-03-26 13:53:48,485 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/A of c54bfce2a8106a499411906befe41bd8 into 87dbc2256ca04fcb83683dc95bd69c4e(size=12.5 K), total size for store is 36.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:48,485 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:48,485 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/A, priority=13, startTime=1679838827999; duration=0sec 2023-03-26 13:53:48,485 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-26 13:53:48,485 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 13:53:48,487 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/B of c54bfce2a8106a499411906befe41bd8 into 873b959ad55f40858dc949cb6d812bde(size=12.5 K), total size for store is 36.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:48,487 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:48,487 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/B, priority=13, startTime=1679838827999; duration=0sec 2023-03-26 13:53:48,487 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-26 13:53:48,488 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/de1fd86df22d44a980017e3248339aac 2023-03-26 13:53:48,488 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73144 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 13:53:48,489 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/C is initiating minor compaction (all files) 2023-03-26 13:53:48,489 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/C in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:48,489 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 6 compacting, 0 eligible, 16 blocking 2023-03-26 13:53:48,489 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 13:53:48,489 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 13:53:48,489 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. because compaction request was cancelled 2023-03-26 13:53:48,489 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:48,489 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/4f6765dd3c8641fd90a27d8e6526cd20, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/f00d83c381c84a59be8c978dcd86a0ab, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e638a5c110284999a0b0206358a2b383, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/cda455e37f044645a987a211e2ff3416, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/8e30cc5b54d0441a95b00eeb56924aa5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/b2dbdb9460fa43c7865c00cc5e7eb885] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=71.4 K 2023-03-26 13:53:48,491 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 4f6765dd3c8641fd90a27d8e6526cd20, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1679838825268 2023-03-26 13:53:48,492 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:48,492 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting f00d83c381c84a59be8c978dcd86a0ab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679838825899 2023-03-26 13:53:48,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:48,492 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:48,493 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/B is initiating minor compaction (all files) 2023-03-26 13:53:48,493 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/B in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:48,493 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/873b959ad55f40858dc949cb6d812bde, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/a05dc142bab644168c737351019ae34c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e15126c2469f4a72ac54484689209501] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=36.3 K 2023-03-26 13:53:48,493 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting e638a5c110284999a0b0206358a2b383, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679838825969 2023-03-26 13:53:48,493 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 873b959ad55f40858dc949cb6d812bde, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1679838826578 2023-03-26 13:53:48,494 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting cda455e37f044645a987a211e2ff3416, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1679838826578 2023-03-26 13:53:48,494 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting a05dc142bab644168c737351019ae34c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1679838827495 2023-03-26 13:53:48,494 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 8e30cc5b54d0441a95b00eeb56924aa5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1679838827495 2023-03-26 13:53:48,494 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e15126c2469f4a72ac54484689209501, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1679838828134 2023-03-26 13:53:48,495 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting b2dbdb9460fa43c7865c00cc5e7eb885, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1679838828134 2023-03-26 13:53:48,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:48,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:48,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:48,500 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:48,500 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:48,500 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:48,500 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:48,501 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,501 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,503 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:48,503 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:48,503 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,503 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,504 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:48,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:48,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:48,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:48,510 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:48,510 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:48,510 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:48,513 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:48,513 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:48,513 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:48,513 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:48,513 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:48,513 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:48,528 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:48,528 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:48,528 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,529 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#B#compaction#67 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:48,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838888540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838888541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838888545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,548 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/a64848af322e495faa0357caeed8e137 2023-03-26 13:53:48,549 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#C#compaction#68 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:48,573 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:48,573 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:48,575 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,575 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:48,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:48,575 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/433589000cc34e38831fa0c212a50de1 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/433589000cc34e38831fa0c212a50de1 2023-03-26 13:53:48,576 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,576 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:48,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:48,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:48,583 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/433589000cc34e38831fa0c212a50de1, entries=250, sequenceid=326, filesize=16.7 K 2023-03-26 13:53:48,585 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/de1fd86df22d44a980017e3248339aac as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/de1fd86df22d44a980017e3248339aac 2023-03-26 13:53:48,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:48,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:48,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40244a5f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:48,591 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:48,591 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,591 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/de1fd86df22d44a980017e3248339aac, entries=150, sequenceid=326, filesize=11.9 K 2023-03-26 13:53:48,592 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,593 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:48,593 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:48,593 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:48,593 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,593 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:48,594 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/a64848af322e495faa0357caeed8e137 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/a64848af322e495faa0357caeed8e137 2023-03-26 13:53:48,598 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:48,599 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:48,599 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:48,599 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:48,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:48,603 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:48,603 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:48,603 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:48,603 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:48,603 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:48,606 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,606 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:48,606 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:48,607 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,609 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:48,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,615 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:48,615 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:48,615 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:48,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:48,615 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:48,615 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:48,615 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:48,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:48,615 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,615 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,615 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,616 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,616 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:48,616 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:48,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:48,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/a64848af322e495faa0357caeed8e137, entries=150, sequenceid=326, filesize=11.9 K 2023-03-26 13:53:48,623 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for c54bfce2a8106a499411906befe41bd8 in 214ms, sequenceid=326, compaction requested=true 2023-03-26 13:53:48,623 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:48,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-26 13:53:48,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-26 13:53:48,623 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-26 13:53:48,635 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/7a59aa5fae2e464dac471c4044c3d957 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/7a59aa5fae2e464dac471c4044c3d957 2023-03-26 13:53:48,645 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c54bfce2a8106a499411906befe41bd8/C of c54bfce2a8106a499411906befe41bd8 into 7a59aa5fae2e464dac471c4044c3d957(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:48,645 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:48,645 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/C, priority=10, startTime=1679838828398; duration=0sec 2023-03-26 13:53:48,646 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-26 13:53:48,646 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:53:48,647 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54272 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:53:48,647 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/A is initiating minor compaction (all files) 2023-03-26 13:53:48,648 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/A in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:48,648 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/87dbc2256ca04fcb83683dc95bd69c4e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/bf6497e677014c8b8c880b3c4e6d018e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/275d709fb2b446ec9d14eee31063ae44, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/433589000cc34e38831fa0c212a50de1] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=53 K 2023-03-26 13:53:48,648 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 87dbc2256ca04fcb83683dc95bd69c4e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1679838826578 2023-03-26 13:53:48,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:48,649 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 13:53:48,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:48,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:48,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:48,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:48,649 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:48,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:48,650 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting bf6497e677014c8b8c880b3c4e6d018e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1679838827495 2023-03-26 13:53:48,652 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 275d709fb2b446ec9d14eee31063ae44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1679838828134 2023-03-26 13:53:48,652 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 433589000cc34e38831fa0c212a50de1, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1679838828290 2023-03-26 13:53:48,690 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#A#compaction#70 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:48,716 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:48,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:48,723 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/f4fa595b83284f52839e314af43d1886 2023-03-26 13:53:48,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:48,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838888747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:48,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:48,760 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:48,760 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:48,760 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:48,760 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:48,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838888750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838888751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679838828723, max timestamp=1679838828750Num uniques 859; 2023-03-26 13:53:48,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 13:53:48,763 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,763 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679838828723, max timestamp=1679838828750Num uniques 859; 2023-03-26 13:53:48,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 13:53:48,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1679838828723, max timestamp=1679838828750Num uniques 859; 2023-03-26 13:53:48,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 13:53:48,767 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,767 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:48,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:48,768 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,768 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:48,770 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/02290a35dfac4f4b96d0ea42399e1fb2 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/02290a35dfac4f4b96d0ea42399e1fb2 2023-03-26 13:53:48,778 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c54bfce2a8106a499411906befe41bd8/A of c54bfce2a8106a499411906befe41bd8 into 02290a35dfac4f4b96d0ea42399e1fb2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:48,778 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:48,778 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/A, priority=12, startTime=1679838828623; duration=0sec 2023-03-26 13:53:48,779 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-26 13:53:48,779 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 3 compacting, 1 eligible, 16 blocking 2023-03-26 13:53:48,779 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 13:53:48,779 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 13:53:48,779 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. because compaction request was cancelled 2023-03-26 13:53:48,780 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-26 13:53:48,780 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 13:53:48,781 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 13:53:48,781 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. because compaction request was cancelled 2023-03-26 13:53:48,781 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 1 store files, 0 compacting, 1 eligible, 16 blocking 2023-03-26 13:53:48,781 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 13:53:48,781 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 13:53:48,781 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. because compaction request was cancelled 2023-03-26 13:53:48,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:48,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:48,796 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:48,796 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:48,796 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:48,823 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:48,824 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:48,824 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:48,824 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:48,824 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:48,824 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/73672e29d916475982245258bc84e08e 2023-03-26 13:53:48,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:48,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:48,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:48,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:48,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:48,827 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,827 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:48,828 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,828 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:48,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:48,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:48,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c505f5c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:48,831 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:48,831 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,833 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,833 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:48,833 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:48,833 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:48,834 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,834 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:48,836 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:48,836 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:48,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838888834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:48,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838888839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:48,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:48,842 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:48,842 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:48,842 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:48,842 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:48,842 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:48,850 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,850 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:48,850 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:48,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:48,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,852 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,854 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:48,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:48,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:48,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:48,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:48,857 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:48,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:48,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,857 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:48,857 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:48,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:48,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:48,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838888864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,865 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838888864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,940 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838888940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:48,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838888943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,957 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:48,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:48,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:48,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:48,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:48,961 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:48,961 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:48,961 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:48,961 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:48,962 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,962 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:48,964 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,964 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,964 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:48,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:48,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:48,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:48,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:48,967 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:48,967 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:48,967 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:48,968 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:48,968 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:48,968 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:48,968 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:48,968 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:48,968 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:48,968 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:48,968 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:48,968 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,968 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:48,968 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:48,970 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:48,970 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:48,971 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,971 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:48,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:48,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:48,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@266c1c91[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:48,973 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:48,973 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,974 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,975 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:48,975 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:48,975 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,975 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:48,975 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:48,976 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,976 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:48,977 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:48,977 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:48,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:48,981 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:48,981 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:48,981 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:48,981 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:48,981 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:48,984 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,984 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:48,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,984 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:48,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:48,985 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,986 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,989 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:48,989 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:48,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:48,989 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:48,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:48,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:48,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:48,989 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,989 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,989 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:48,989 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:48,989 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,989 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:48,989 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:48,990 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:48,990 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:48,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:49,014 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/d4be37d176fd48d8854ff1ac41ac43d6 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/d4be37d176fd48d8854ff1ac41ac43d6 2023-03-26 13:53:49,022 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/B of c54bfce2a8106a499411906befe41bd8 into d4be37d176fd48d8854ff1ac41ac43d6(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:49,023 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:49,023 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/B, priority=13, startTime=1679838828398; duration=0sec 2023-03-26 13:53:49,023 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:49,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:49,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838889056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:49,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838889071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,074 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:49,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838889072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,090 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:49,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:49,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:49,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:49,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:49,104 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:49,104 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:49,104 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:49,104 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:49,104 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,105 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,111 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,111 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:49,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:49,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:49,111 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,112 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:49,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:49,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,113 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:49,113 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:49,113 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:49,113 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:49,113 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:49,113 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:49,114 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:49,114 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:49,114 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:49,114 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:49,114 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:49,116 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,116 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:49,117 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,117 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:49,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:49,120 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:49,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16d79a86[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:49,120 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,121 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,122 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:49,122 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:49,122 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:49,122 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,122 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:49,123 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,123 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:49,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:49,123 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:49,124 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:49,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:49,128 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:49,128 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:49,128 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:49,128 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:49,128 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:49,131 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,131 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:49,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,131 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:49,131 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,138 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:49,138 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:49,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:49,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:49,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:49,138 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:49,138 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,138 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,138 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,138 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,138 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,138 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:49,138 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:49,138 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:49,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:49,138 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,143 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:49,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838889143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:49,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838889148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,239 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:49,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:49,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:49,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:49,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:49,244 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:49,244 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:49,244 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:49,244 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:49,244 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,244 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,246 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:49,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:49,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:49,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,247 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,247 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,248 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:49,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:49,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:49,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:49,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:49,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:49,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:49,250 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:49,250 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:49,250 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:49,250 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:49,250 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:49,250 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:49,250 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:49,250 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,250 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:49,250 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:49,252 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/8020f124f1a24a64b28d9940e6ce87e0 2023-03-26 13:53:49,252 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:49,254 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:49,256 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,256 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:49,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:49,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74c35cb1[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:49,258 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:49,258 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,261 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,261 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:49,261 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:49,261 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:49,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:49,262 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/f4fa595b83284f52839e314af43d1886 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/f4fa595b83284f52839e314af43d1886 2023-03-26 13:53:49,263 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,263 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:49,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:49,263 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:49,264 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:49,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:49,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:49,268 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:49,268 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:49,268 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:49,268 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:49,268 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:49,270 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/f4fa595b83284f52839e314af43d1886, entries=150, sequenceid=341, filesize=11.9 K 2023-03-26 13:53:49,270 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,270 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,270 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:49,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:49,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:49,271 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,272 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/73672e29d916475982245258bc84e08e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/73672e29d916475982245258bc84e08e 2023-03-26 13:53:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,274 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:49,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:49,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:49,279 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:49,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:49,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,279 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:49,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:49,280 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:49,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:49,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:49,284 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/73672e29d916475982245258bc84e08e, entries=100, sequenceid=341, filesize=9.5 K 2023-03-26 13:53:49,285 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/8020f124f1a24a64b28d9940e6ce87e0 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/8020f124f1a24a64b28d9940e6ce87e0 2023-03-26 13:53:49,302 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/8020f124f1a24a64b28d9940e6ce87e0, entries=100, sequenceid=341, filesize=9.5 K 2023-03-26 13:53:49,303 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for c54bfce2a8106a499411906befe41bd8 in 654ms, sequenceid=341, compaction requested=true 2023-03-26 13:53:49,303 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:49,304 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:49,304 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-26 13:53:49,304 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:49,304 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:49,304 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:49,305 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 13:53:49,305 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 13:53:49,305 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. because compaction request was cancelled 2023-03-26 13:53:49,305 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:49,305 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:49,305 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/B is initiating minor compaction (all files) 2023-03-26 13:53:49,305 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/B in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:49,305 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/d4be37d176fd48d8854ff1ac41ac43d6, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/de1fd86df22d44a980017e3248339aac, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/73672e29d916475982245258bc84e08e] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=34.0 K 2023-03-26 13:53:49,306 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting d4be37d176fd48d8854ff1ac41ac43d6, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1679838828134 2023-03-26 13:53:49,306 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:49,306 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/C is initiating minor compaction (all files) 2023-03-26 13:53:49,306 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting de1fd86df22d44a980017e3248339aac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1679838828296 2023-03-26 13:53:49,306 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/C in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:49,307 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/7a59aa5fae2e464dac471c4044c3d957, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/a64848af322e495faa0357caeed8e137, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/8020f124f1a24a64b28d9940e6ce87e0] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=34.0 K 2023-03-26 13:53:49,307 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 73672e29d916475982245258bc84e08e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1679838828428 2023-03-26 13:53:49,307 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 7a59aa5fae2e464dac471c4044c3d957, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1679838828134 2023-03-26 13:53:49,308 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting a64848af322e495faa0357caeed8e137, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1679838828296 2023-03-26 13:53:49,309 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 8020f124f1a24a64b28d9940e6ce87e0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1679838828428 2023-03-26 13:53:49,321 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#B#compaction#73 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:49,324 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#C#compaction#74 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:49,337 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/747e2e608f3a4c64912b9841bd2ef98c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/747e2e608f3a4c64912b9841bd2ef98c 2023-03-26 13:53:49,345 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/C of c54bfce2a8106a499411906befe41bd8 into 747e2e608f3a4c64912b9841bd2ef98c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:49,345 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:49,345 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/C, priority=13, startTime=1679838829304; duration=0sec 2023-03-26 13:53:49,345 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:49,347 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/95664e0ce60b4603990b777853936085 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/95664e0ce60b4603990b777853936085 2023-03-26 13:53:49,353 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/B of c54bfce2a8106a499411906befe41bd8 into 95664e0ce60b4603990b777853936085(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:49,353 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:49,353 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/B, priority=13, startTime=1679838829304; duration=0sec 2023-03-26 13:53:49,353 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:49,380 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:49,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:49,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:49,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:49,383 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-26 13:53:49,383 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:49,383 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:49,383 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:49,384 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:49,384 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:49,384 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:49,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:49,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:49,384 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:49,384 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:49,384 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:49,384 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:49,385 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,385 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,390 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,390 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:49,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:49,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:49,390 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,390 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:49,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:49,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:49,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:49,396 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:49,396 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:49,396 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:49,396 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:49,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:49,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838889397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,401 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:49,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838889398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,412 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:49,412 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:49,414 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:49,418 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,418 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/8ad2c36079404498a0395167d936a767 2023-03-26 13:53:49,418 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,418 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:49,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:49,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ae747ca[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:49,419 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:49,419 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,420 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,420 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:49,420 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:49,420 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:49,421 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,421 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:49,422 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,422 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:49,422 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:49,422 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:49,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:49,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:49,428 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:49,428 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:49,428 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:49,428 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:49,428 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:49,431 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:49,432 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,433 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,433 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:49,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:49,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,434 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:49,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:49,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:49,437 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:49,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:49,438 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:49,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:49,438 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:49,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:49,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:49,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:49,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838889444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:49,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838889450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:49,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838889501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:49,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838889502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,538 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:49,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:49,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:49,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:49,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:49,542 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:49,542 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:49,542 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:49,542 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:49,543 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,543 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,544 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,544 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:49,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:49,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:49,545 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,545 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:49,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:49,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:49,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:49,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:49,546 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:49,546 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:49,546 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:49,546 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:49,546 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:49,546 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:49,546 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:49,547 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:49,547 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:49,547 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:49,547 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:49,547 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,547 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:49,547 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:49,548 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,548 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:49,549 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,549 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:49,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:49,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ca94493[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:49,550 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:49,550 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,553 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:49,553 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:49,553 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,553 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:49,553 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:49,555 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,555 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:49,555 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:49,555 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:49,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:49,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:49,558 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:49,558 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:49,558 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:49,558 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:49,558 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:49,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:49,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838889559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,560 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,560 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:49,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,560 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:49,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,561 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,562 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,571 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:49,571 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:49,571 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:49,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:49,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:49,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:49,571 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:49,571 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:49,571 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:49,571 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,572 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,572 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,572 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,572 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,572 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,672 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:49,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:49,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:49,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:49,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:49,676 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:49,676 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:49,676 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:49,676 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:49,677 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,677 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,678 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:49,678 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:49,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:49,679 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,679 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:49,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:49,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:49,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:49,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:49,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:49,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:49,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:49,681 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:49,681 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:49,681 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:49,681 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:49,682 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:49,682 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:49,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:49,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:49,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:49,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:49,686 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,686 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:49,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:49,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:49,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1600ea2a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:49,688 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:49,688 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,689 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,689 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:49,689 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:49,689 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:49,690 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,690 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:49,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,692 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:49,692 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:49,692 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:49,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:49,695 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:49,695 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:49,695 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:49,695 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:49,695 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:49,698 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,698 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:49,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,698 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:49,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:49,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,700 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,701 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:49,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838889704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:49,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39056 deadline: 1679838889705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,705 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,706 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,706 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,706 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,706 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,706 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,706 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:49,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:49,705 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:49,706 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:49,706 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:49,706 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:49,706 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:49,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:49,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:49,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:49,806 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:49,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:49,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:49,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:49,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:49,812 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:49,813 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:49,813 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:49,813 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:49,813 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,813 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,814 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,815 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:49,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:49,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:49,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:49,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:49,818 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:49,818 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:49,818 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:49,820 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:49,820 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:49,820 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:49,820 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:49,820 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:49,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,822 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:49,823 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,823 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:49,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:49,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78480a8f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:49,825 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:49,825 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,827 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,827 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:49,827 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:49,827 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:49,827 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,827 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:49,830 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:49,830 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:49,831 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:49,831 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:49,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:49,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:49,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:49,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:49,836 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:49,836 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:49,836 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:49,836 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:49,836 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:49,839 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/9e2ca7cf4e7a4347972eebcd65633320 2023-03-26 13:53:49,839 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:49,839 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,839 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:49,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:49,845 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:49,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:49,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:49,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,850 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:49,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,861 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:49,861 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:49,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:49,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:49,862 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,862 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,862 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,862 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,862 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:49,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:49,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:49,862 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:49,862 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:49,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:49,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:49,862 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,862 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:49,862 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:49,881 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=380 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/9801fac668ff4fa1a948161982bf7221 2023-03-26 13:53:49,889 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/8ad2c36079404498a0395167d936a767 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/8ad2c36079404498a0395167d936a767 2023-03-26 13:53:49,898 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/8ad2c36079404498a0395167d936a767, entries=150, sequenceid=380, filesize=11.9 K 2023-03-26 13:53:49,899 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/9e2ca7cf4e7a4347972eebcd65633320 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/9e2ca7cf4e7a4347972eebcd65633320 2023-03-26 13:53:49,912 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/9e2ca7cf4e7a4347972eebcd65633320, entries=150, sequenceid=380, filesize=11.9 K 2023-03-26 13:53:49,914 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/9801fac668ff4fa1a948161982bf7221 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/9801fac668ff4fa1a948161982bf7221 2023-03-26 13:53:49,924 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/9801fac668ff4fa1a948161982bf7221, entries=150, sequenceid=380, filesize=11.9 K 2023-03-26 13:53:49,925 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for c54bfce2a8106a499411906befe41bd8 in 542ms, sequenceid=380, compaction requested=true 2023-03-26 13:53:49,925 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:49,926 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:49,926 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:49,926 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:49,926 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-26 13:53:49,926 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:49,927 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 13:53:49,927 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 13:53:49,927 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. because compaction request was cancelled 2023-03-26 13:53:49,927 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-26 13:53:49,927 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37338 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:49,927 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/A is initiating minor compaction (all files) 2023-03-26 13:53:49,928 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/A in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:49,928 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/02290a35dfac4f4b96d0ea42399e1fb2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/f4fa595b83284f52839e314af43d1886, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/8ad2c36079404498a0395167d936a767] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=36.5 K 2023-03-26 13:53:49,928 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 02290a35dfac4f4b96d0ea42399e1fb2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1679838828296 2023-03-26 13:53:49,929 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 13:53:49,929 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 13:53:49,929 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. because compaction request was cancelled 2023-03-26 13:53:49,930 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting f4fa595b83284f52839e314af43d1886, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1679838828428 2023-03-26 13:53:49,931 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 8ad2c36079404498a0395167d936a767, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1679838828836 2023-03-26 13:53:49,945 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#A#compaction#78 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:49,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:49,952 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 13:53:49,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:49,953 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:49,953 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:49,953 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:49,953 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:49,953 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:49,963 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:49,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:49,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:49,986 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/82366c9d993b4675a23ba8a0485f1674 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/82366c9d993b4675a23ba8a0485f1674 2023-03-26 13:53:49,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:49,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:49,989 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:49,989 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:49,989 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:49,989 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:49,990 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,990 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,993 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:49,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:49,993 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,995 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:49,995 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:49,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:49,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:50,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:50,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:50,001 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:50,001 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:50,001 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:50,013 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/A of c54bfce2a8106a499411906befe41bd8 into 82366c9d993b4675a23ba8a0485f1674(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:50,013 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:50,014 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/A, priority=13, startTime=1679838829926; duration=0sec 2023-03-26 13:53:50,014 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:50,017 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:50,017 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:50,018 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:50,018 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:50,018 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:50,018 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:50,018 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:50,020 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,021 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:50,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3851db5b[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:50,023 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:50,023 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,025 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,025 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:50,026 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:50,026 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,026 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:50,030 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:50,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:50,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:50,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:50,034 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:50,034 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:50,034 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:50,034 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:50,034 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:50,039 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,039 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:50,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,039 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,041 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838830023, max timestamp=1679838830038Num uniques 859; 2023-03-26 13:53:50,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838830023, max timestamp=1679838830038Num uniques 859; 2023-03-26 13:53:50,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838830023, max timestamp=1679838830038Num uniques 859; 2023-03-26 13:53:50,044 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:53:50,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:53:50,044 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:53:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,048 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:50,048 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:50,048 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:50,048 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:50,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:50,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:50,049 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:50,049 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:50,048 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,049 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,049 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,049 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,049 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,049 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:50,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:50,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:50,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:50,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838890075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:50,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838890077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,082 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:50,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39064 deadline: 1679838890079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:50,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838890082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,149 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:50,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:50,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:50,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:50,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:50,156 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:50,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:50,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:50,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:50,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,160 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:50,160 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,161 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,161 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:50,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,162 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:50,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:50,162 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:50,162 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:50,162 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:50,164 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:50,164 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:50,164 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:50,164 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:50,164 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:50,164 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:50,164 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:50,164 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:50,164 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,164 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:50,164 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:50,167 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,167 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,167 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,167 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:50,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eaa8d1e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:50,170 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:50,170 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,172 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,172 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:50,172 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:50,172 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:50,172 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,172 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:50,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:50,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:50,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:50,175 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:50,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:50,179 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:50,179 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:50,179 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:50,179 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:50,179 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:50,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:50,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838890183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:50,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838890183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,190 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,190 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:50,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838890185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,190 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:50,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:50,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:50,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,191 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:50,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838890187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,194 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,200 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,200 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,200 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,201 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,201 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,201 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,201 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,201 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:50,201 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:50,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:50,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:50,201 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:50,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:50,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:50,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:50,203 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:50,203 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:50,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:50,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:50,303 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:50,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:50,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:50,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:50,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:50,318 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:50,318 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:50,319 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:50,319 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:50,319 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,319 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,321 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:50,321 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:50,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:50,321 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,321 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:50,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:50,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:50,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:50,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:50,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:50,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:50,332 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:50,332 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:50,332 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:50,332 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:50,332 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,333 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:50,333 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:50,334 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,335 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,336 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,336 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:50,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f6a6f5b[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:50,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:50,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,338 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,339 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:50,339 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:50,339 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:50,339 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,339 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:50,342 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:50,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:50,347 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:50,347 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:50,347 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:50,347 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:50,347 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:50,349 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,349 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,350 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:50,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,351 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,352 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,355 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:50,355 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:50,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:50,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:50,355 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:50,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:50,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:50,356 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,356 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,356 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,356 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,356 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,356 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,356 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,356 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:50,356 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:50,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:50,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:50,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:50,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838890387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:50,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838890388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:50,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838890391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,394 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:50,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838890394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,421 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/7d36509044cc42a783efb4a3649705d9 2023-03-26 13:53:50,456 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:50,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:50,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:50,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:50,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:50,467 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:50,467 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:50,467 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:50,467 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:50,468 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,468 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,470 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:50,470 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,471 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,471 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:50,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,471 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:50,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:50,483 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:50,483 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/2207660a7b3547589fe4d50fb98653c4 2023-03-26 13:53:50,483 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:50,483 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:50,491 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:50,491 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:50,492 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:50,492 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:50,492 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:50,492 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:50,492 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:50,492 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:50,492 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,492 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:50,492 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:50,497 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,498 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,500 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,500 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:50,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:50,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67ce1a4b[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:50,509 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:50,509 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,512 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,512 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:50,512 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:50,512 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:50,512 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,513 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:50,514 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:50,514 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,514 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,514 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:50,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:50,519 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:50,519 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:50,519 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:50,519 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:50,519 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:50,521 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,521 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:50,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:50,522 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,523 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,525 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,530 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,530 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,530 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:50,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:50,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:50,530 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:50,530 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:50,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:50,530 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:50,530 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:50,530 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:50,530 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,531 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,531 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,531 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:50,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:50,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838890567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,631 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:50,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:50,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:50,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:50,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:50,634 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:50,634 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:50,634 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:50,634 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:50,635 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,635 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,637 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,637 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:50,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:50,637 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,637 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:50,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:50,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:50,638 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:50,638 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:50,638 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:50,639 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:50,639 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:50,639 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:50,639 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:50,639 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:50,639 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:50,639 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:50,639 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:50,639 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,639 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:50,639 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:50,641 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,641 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,642 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,642 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:50,643 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:50,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51f39eea[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:50,643 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,648 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,648 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:50,649 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:50,649 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:50,649 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,649 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:50,650 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:50,650 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,650 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:50,650 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:50,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:50,653 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:50,653 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:50,653 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:50,653 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:50,653 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:50,655 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,655 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,655 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,657 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,659 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,662 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,662 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:50,662 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:50,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:50,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:50,662 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,662 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:50,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:50,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:50,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:50,662 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,662 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,662 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,662 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,662 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,662 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:50,662 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:50,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:50,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:50,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:50,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838890691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:50,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838890692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:50,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838890695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:50,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838890696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,762 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:50,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:50,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:50,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:50,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:50,767 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:50,767 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:50,767 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:50,767 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:50,767 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,767 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,770 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,770 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:50,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:50,770 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,770 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:50,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:50,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:50,772 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:50,772 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:50,772 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:50,772 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:50,772 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:50,774 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,774 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,775 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,775 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:50,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b2b3bb7[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:50,776 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:50,776 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,778 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,778 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:50,778 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:50,778 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:50,778 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,778 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:50,779 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:50,779 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,780 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,780 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:50,784 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:50,784 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:50,784 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:50,784 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:50,784 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:50,787 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,787 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,787 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,788 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,789 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,791 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,792 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,792 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,792 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,792 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:50,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:50,792 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:50,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:50,792 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:50,792 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:50,792 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:50,792 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:50,792 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,792 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:50,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:50,893 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:50,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:50,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:50,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:50,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:50,908 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:50,908 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:50,908 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:50,908 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:50,912 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,912 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,913 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:50,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:50,914 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,914 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:50,916 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:50,916 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:50,916 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:50,916 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:50,916 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:50,920 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:50,920 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:50,920 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:50,920 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:50,920 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:50,924 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:50,926 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47b7853e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:50,926 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,927 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=391 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/b5281181440d43b5812b1d14b3893945 2023-03-26 13:53:50,928 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:50,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:50,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:50,929 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,929 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:50,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:50,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:50,930 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,933 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/7d36509044cc42a783efb4a3649705d9 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/7d36509044cc42a783efb4a3649705d9 2023-03-26 13:53:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:50,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:50,934 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:50,934 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:50,934 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:50,934 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:50,934 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:50,936 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,936 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:50,936 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:50,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,938 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/7d36509044cc42a783efb4a3649705d9, entries=200, sequenceid=391, filesize=14.3 K 2023-03-26 13:53:50,938 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,941 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/2207660a7b3547589fe4d50fb98653c4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/2207660a7b3547589fe4d50fb98653c4 2023-03-26 13:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,941 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:50,944 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:50,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:50,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:50,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:50,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:50,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:50,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:50,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:50,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:50,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:50,945 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:50,945 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:50,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:50,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:50,948 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/2207660a7b3547589fe4d50fb98653c4, entries=150, sequenceid=391, filesize=11.9 K 2023-03-26 13:53:50,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/b5281181440d43b5812b1d14b3893945 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/b5281181440d43b5812b1d14b3893945 2023-03-26 13:53:50,954 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/b5281181440d43b5812b1d14b3893945, entries=150, sequenceid=391, filesize=11.9 K 2023-03-26 13:53:50,955 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for c54bfce2a8106a499411906befe41bd8 in 1003ms, sequenceid=391, compaction requested=true 2023-03-26 13:53:50,956 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:50,956 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:50,956 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-26 13:53:50,956 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:50,956 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:50,956 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:50,964 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 13:53:50,964 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 13:53:50,964 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. because compaction request was cancelled 2023-03-26 13:53:50,964 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:50,965 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:50,965 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/B is initiating minor compaction (all files) 2023-03-26 13:53:50,965 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/B in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:50,965 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/95664e0ce60b4603990b777853936085, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/9e2ca7cf4e7a4347972eebcd65633320, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/2207660a7b3547589fe4d50fb98653c4] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=36.5 K 2023-03-26 13:53:50,966 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 95664e0ce60b4603990b777853936085, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1679838828296 2023-03-26 13:53:50,966 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:50,966 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/C is initiating minor compaction (all files) 2023-03-26 13:53:50,966 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/C in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:50,966 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/747e2e608f3a4c64912b9841bd2ef98c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/9801fac668ff4fa1a948161982bf7221, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/b5281181440d43b5812b1d14b3893945] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=36.5 K 2023-03-26 13:53:50,966 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 9e2ca7cf4e7a4347972eebcd65633320, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1679838828836 2023-03-26 13:53:50,967 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 747e2e608f3a4c64912b9841bd2ef98c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1679838828296 2023-03-26 13:53:50,968 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 9801fac668ff4fa1a948161982bf7221, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1679838828836 2023-03-26 13:53:50,968 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting b5281181440d43b5812b1d14b3893945, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1679838829396 2023-03-26 13:53:50,969 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 2207660a7b3547589fe4d50fb98653c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1679838829396 2023-03-26 13:53:50,977 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#C#compaction#82 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:50,985 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#B#compaction#83 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:51,015 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/c6471d2600ac428c888b0ad6059aa92a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/c6471d2600ac428c888b0ad6059aa92a 2023-03-26 13:53:51,016 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/ac6a9313f0ba4ec8b399cadce02f489a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/ac6a9313f0ba4ec8b399cadce02f489a 2023-03-26 13:53:51,023 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/C of c54bfce2a8106a499411906befe41bd8 into ac6a9313f0ba4ec8b399cadce02f489a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:51,023 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:51,023 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/C, priority=13, startTime=1679838830956; duration=0sec 2023-03-26 13:53:51,023 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:51,041 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/B of c54bfce2a8106a499411906befe41bd8 into c6471d2600ac428c888b0ad6059aa92a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:51,041 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:51,041 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/B, priority=13, startTime=1679838830956; duration=0sec 2023-03-26 13:53:51,041 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:51,045 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:51,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:51,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:51,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:51,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:51,052 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:51,052 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:51,052 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:51,052 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:51,052 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,053 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,055 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:51,055 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:51,056 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,056 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:51,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:51,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:51,057 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:51,057 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:51,057 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:51,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:51,060 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:51,060 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:51,060 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-26 13:53:51,060 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:51,060 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:51,061 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:51,061 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:51,061 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:51,061 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:51,086 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/74eb06fc70024b138ed307c0f0cb2bed 2023-03-26 13:53:51,112 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/24f5338722a8472a9fe223e7fd6efbbd 2023-03-26 13:53:51,146 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/56de2004a1114c67879dc2174e3e9e59 2023-03-26 13:53:51,154 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/74eb06fc70024b138ed307c0f0cb2bed as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/74eb06fc70024b138ed307c0f0cb2bed 2023-03-26 13:53:51,163 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/74eb06fc70024b138ed307c0f0cb2bed, entries=150, sequenceid=430, filesize=11.9 K 2023-03-26 13:53:51,165 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/24f5338722a8472a9fe223e7fd6efbbd as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/24f5338722a8472a9fe223e7fd6efbbd 2023-03-26 13:53:51,170 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/24f5338722a8472a9fe223e7fd6efbbd, entries=150, sequenceid=430, filesize=11.9 K 2023-03-26 13:53:51,171 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/56de2004a1114c67879dc2174e3e9e59 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/56de2004a1114c67879dc2174e3e9e59 2023-03-26 13:53:51,179 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/56de2004a1114c67879dc2174e3e9e59, entries=150, sequenceid=430, filesize=11.9 K 2023-03-26 13:53:51,180 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for c54bfce2a8106a499411906befe41bd8 in 120ms, sequenceid=430, compaction requested=true 2023-03-26 13:53:51,180 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:51,180 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:51,180 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:51,180 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:51,180 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:51,188 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,189 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:51,189 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,189 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:51,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:51,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50bcd3c1[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:51,192 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:51,192 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,195 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,195 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:51,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:51,201 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:51,201 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:51,201 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:51,201 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:51,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:51,212 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:51,212 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:51,212 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:51,212 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:51,212 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:51,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:51,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:51,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:51,215 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:51,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:51,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:51,216 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:51,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,217 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:51,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,220 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:51,220 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:51,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:51,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:51,220 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,221 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,221 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:51,221 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,221 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:51,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:51,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:51,221 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:51,221 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:51,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:51,221 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:51,222 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 13:53:51,222 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,222 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,223 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:51,223 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:51,223 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:51,223 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:51,223 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:51,223 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:51,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:51,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838891253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,262 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:51,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838891258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838831248, max timestamp=1679838831258Num uniques 859; 2023-03-26 13:53:51,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:53:51,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838831248, max timestamp=1679838831258Num uniques 859; 2023-03-26 13:53:51,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:53:51,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838831248, max timestamp=1679838831258Num uniques 859; 2023-03-26 13:53:51,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:53:51,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:51,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838891311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:51,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838891311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,321 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:51,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:51,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:51,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:51,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:51,325 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:51,325 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:51,325 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:51,325 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:51,325 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,325 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,326 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:51,326 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,327 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,327 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:51,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:51,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:51,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:51,328 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:51,328 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:51,328 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:51,336 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:51,336 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:51,336 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:51,336 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:51,336 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:51,336 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:51,336 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:51,336 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:51,336 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,336 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:51,336 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:51,338 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,338 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:51,339 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,339 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:51,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:51,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:51,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:51,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1df513f2[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:51,341 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:51,341 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,343 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,343 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:51,343 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:51,343 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:51,343 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,344 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:51,348 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:51,348 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,348 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:51,348 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:51,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:51,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:51,351 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:51,351 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:51,351 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:51,351 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:51,351 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:51,354 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,354 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:51,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,354 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:51,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:51,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:51,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,356 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:51,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,358 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,361 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,361 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,362 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,362 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:51,362 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:51,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:51,362 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,362 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,362 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,362 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,362 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:51,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:51,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:51,362 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:51,362 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:51,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:51,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:51,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:51,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838891361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,365 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:51,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838891363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:51,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838891416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:51,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838891416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,462 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:51,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:51,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:51,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:51,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:51,466 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:51,466 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:51,466 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:51,466 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:51,467 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,467 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,468 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:51,468 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:51,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:51,469 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,469 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:51,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:51,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:51,470 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:51,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:51,470 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:51,470 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:51,470 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:51,470 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:51,470 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:51,470 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:51,470 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:51,471 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:51,471 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:51,471 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:51,471 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:51,471 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,471 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:51,471 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:51,473 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,473 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:51,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:51,474 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,474 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:51,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:51,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:51,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:51,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:51,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f916578[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:51,475 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:51,475 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,477 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,477 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:51,477 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:51,477 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,477 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:51,477 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:51,478 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:51,478 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:51,478 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,478 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:51,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:51,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:51,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:51,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:51,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:51,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:51,481 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:51,481 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:51,481 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:51,481 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:51,481 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:51,482 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,482 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,482 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:51,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:51,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:51,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:51,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:51,483 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:51,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:51,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,484 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,487 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,487 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:51,487 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:51,487 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:51,488 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:51,487 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:51,488 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:51,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:51,488 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,488 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,488 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:51,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:51,488 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:51,488 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:51,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:51,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838891567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:51,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838891567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,588 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:51,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:51,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:51,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:51,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:51,594 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:51,594 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:51,595 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:51,595 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:51,595 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,595 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,598 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:51,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:51,598 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,600 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,600 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,600 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:51,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:51,609 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:51,610 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:51,610 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:51,610 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:51,612 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:51,612 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:51,612 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:51,612 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:51,612 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:51,612 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:51,612 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:51,612 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:51,612 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,612 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:51,613 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:51,616 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,616 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:51,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,617 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:51,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:51,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:51,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:51,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74f49fc1[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:51,619 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:51,619 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:51,622 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,622 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:51,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:51,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838891621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:51,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838891622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,623 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:51,624 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:51,624 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:51,624 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:51,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:51,626 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:51,627 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:51,627 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:51,627 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:51,627 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:51,629 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,629 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:51,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:51,629 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:51,630 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,632 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,640 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,641 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,641 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,641 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,641 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:51,641 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:51,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:51,641 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,641 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,641 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:51,641 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:51,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:51,641 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:51,641 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:51,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:51,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:51,677 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/3e135763009a4fd39321f6318cd61e04 2023-03-26 13:53:51,741 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:51,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:51,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:51,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:51,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:51,747 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:51,747 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:51,747 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:51,747 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:51,747 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,747 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,749 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,749 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:51,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:51,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:51,749 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,749 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:51,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:51,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:51,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:51,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:51,750 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:51,752 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:51,752 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:51,752 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:51,752 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:51,752 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:51,752 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:51,752 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:51,752 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:51,752 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,752 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:51,752 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:51,754 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,754 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:51,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:51,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:51,755 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,755 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:51,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:51,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:51,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:51,756 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:51,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42606845[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:51,756 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,758 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,758 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:51,758 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:51,758 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,758 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:51,758 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:51,761 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,761 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:51,761 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:51,761 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:51,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:51,764 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:51,764 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:51,764 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:51,764 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:51,764 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:51,766 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,766 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:51,766 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:51,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:51,768 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,769 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,772 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,772 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,772 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,772 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:51,772 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,773 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:51,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:51,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:51,773 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,773 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:51,773 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,773 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:51,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:51,773 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:51,773 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:51,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:51,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:51,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838891870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,887 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:51,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:51,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838891887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:51,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:51,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:51,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:51,895 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:51,895 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:51,895 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:51,895 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:51,897 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,897 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,898 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:51,898 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:51,899 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,899 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:51,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:51,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:51,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:51,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:51,904 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:51,904 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:51,904 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:51,904 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:51,904 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:51,904 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:51,904 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:51,904 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:51,904 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,904 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:51,904 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:51,908 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,908 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:51,908 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,908 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:51,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:51,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:51,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77bf9978[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:51,911 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:51,911 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,912 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,912 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:51,912 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:51,912 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:51,913 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,913 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:51,914 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:51,914 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:51,914 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:51,914 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:51,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:51,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:51,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:51,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:51,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:51,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:51,921 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:51,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:51,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:51,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:51,921 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:51,923 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:51,924 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,924 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:51,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:51,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,926 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:51,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:51,928 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:51,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838891925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:51,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838891925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,932 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,932 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,932 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:51,932 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:51,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:51,932 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:51,932 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:51,932 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:51,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:51,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:51,932 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,932 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:51,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:51,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:51,932 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:51,932 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:51,932 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:52,033 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:52,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:52,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:52,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:52,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:52,036 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:52,036 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:52,036 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:52,036 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:52,037 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,037 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,038 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,038 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:52,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:52,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:52,039 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,039 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:52,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:52,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:52,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:52,040 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:52,040 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:52,040 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:52,040 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:52,041 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:52,041 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:52,041 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:52,041 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:52,041 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:52,041 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:52,041 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:52,041 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,041 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:52,041 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:52,043 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:52,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:52,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:52,045 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:52,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73018852[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:52,045 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:52,047 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:52,047 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,047 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:52,049 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:52,049 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:52,050 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:52,050 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:52,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:52,053 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:52,053 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:52,053 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:52,053 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:52,053 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:52,055 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:52,055 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,055 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:52,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:52,056 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,058 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,063 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,063 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,063 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,063 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,063 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:52,063 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:52,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:52,064 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:52,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:52,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:52,063 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,064 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:52,064 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,064 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:52,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:52,064 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:52,098 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/95b17d4c479d479cac7ba3b3b6e0f824 2023-03-26 13:53:52,119 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=442 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/3bf19f93ae884aff985300578a1f21ec 2023-03-26 13:53:52,132 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/3e135763009a4fd39321f6318cd61e04 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/3e135763009a4fd39321f6318cd61e04 2023-03-26 13:53:52,141 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/3e135763009a4fd39321f6318cd61e04, entries=200, sequenceid=442, filesize=14.3 K 2023-03-26 13:53:52,142 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/95b17d4c479d479cac7ba3b3b6e0f824 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/95b17d4c479d479cac7ba3b3b6e0f824 2023-03-26 13:53:52,148 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/95b17d4c479d479cac7ba3b3b6e0f824, entries=150, sequenceid=442, filesize=11.9 K 2023-03-26 13:53:52,150 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/3bf19f93ae884aff985300578a1f21ec as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/3bf19f93ae884aff985300578a1f21ec 2023-03-26 13:53:52,156 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/3bf19f93ae884aff985300578a1f21ec, entries=150, sequenceid=442, filesize=11.9 K 2023-03-26 13:53:52,157 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for c54bfce2a8106a499411906befe41bd8 in 935ms, sequenceid=442, compaction requested=true 2023-03-26 13:53:52,157 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:52,157 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:52,157 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:53:52,157 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:52,158 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:52,158 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:52,159 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:52,159 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/B is initiating minor compaction (all files) 2023-03-26 13:53:52,159 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/B in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:52,159 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/c6471d2600ac428c888b0ad6059aa92a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/24f5338722a8472a9fe223e7fd6efbbd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/95b17d4c479d479cac7ba3b3b6e0f824] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=36.6 K 2023-03-26 13:53:52,159 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54510 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:53:52,160 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/A is initiating minor compaction (all files) 2023-03-26 13:53:52,160 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting c6471d2600ac428c888b0ad6059aa92a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1679838829396 2023-03-26 13:53:52,160 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/A in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:52,160 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/82366c9d993b4675a23ba8a0485f1674, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/7d36509044cc42a783efb4a3649705d9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/74eb06fc70024b138ed307c0f0cb2bed, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/3e135763009a4fd39321f6318cd61e04] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=53.2 K 2023-03-26 13:53:52,160 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 24f5338722a8472a9fe223e7fd6efbbd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1679838830060 2023-03-26 13:53:52,160 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 82366c9d993b4675a23ba8a0485f1674, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=380, earliestPutTs=1679838828836 2023-03-26 13:53:52,161 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 95b17d4c479d479cac7ba3b3b6e0f824, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1679838831209 2023-03-26 13:53:52,161 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 7d36509044cc42a783efb4a3649705d9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1679838829394 2023-03-26 13:53:52,161 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 74eb06fc70024b138ed307c0f0cb2bed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1679838830060 2023-03-26 13:53:52,162 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 3e135763009a4fd39321f6318cd61e04, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1679838831209 2023-03-26 13:53:52,164 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:52,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:52,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:52,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:52,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:52,172 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:52,172 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:52,172 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:52,172 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:52,173 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,173 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,175 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:52,175 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:52,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,176 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,176 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:52,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:52,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:52,177 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#B#compaction#90 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:52,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:52,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:52,180 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:52,180 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:52,180 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:52,182 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#A#compaction#91 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:52,182 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:52,182 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:52,182 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:52,182 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:52,183 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=120.76 KB heapSize=345.14 KB 2023-03-26 13:53:52,183 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:52,183 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:52,183 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:52,183 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:52,183 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:52,183 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:52,204 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/3cd5368930bf447b83b4e0856113ac15 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/3cd5368930bf447b83b4e0856113ac15 2023-03-26 13:53:52,212 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/B of c54bfce2a8106a499411906befe41bd8 into 3cd5368930bf447b83b4e0856113ac15(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:52,212 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:52,212 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/B, priority=13, startTime=1679838832157; duration=0sec 2023-03-26 13:53:52,212 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:52,213 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:52,214 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:52,214 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/C is initiating minor compaction (all files) 2023-03-26 13:53:52,214 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/C in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:52,214 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/ac6a9313f0ba4ec8b399cadce02f489a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/56de2004a1114c67879dc2174e3e9e59, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/3bf19f93ae884aff985300578a1f21ec] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=36.6 K 2023-03-26 13:53:52,215 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting ac6a9313f0ba4ec8b399cadce02f489a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=391, earliestPutTs=1679838829396 2023-03-26 13:53:52,215 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 56de2004a1114c67879dc2174e3e9e59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1679838830060 2023-03-26 13:53:52,216 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 3bf19f93ae884aff985300578a1f21ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1679838831209 2023-03-26 13:53:52,217 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/1fa60de424bd434caabbed8657aec23d 2023-03-26 13:53:52,228 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#C#compaction#94 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:52,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:52,394 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:52,406 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:52,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838892405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,409 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:52,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838892406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:52,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838892434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:52,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838892434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,509 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:52,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838892507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:52,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838892511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,583 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:52,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838892582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,586 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., hostname=jenkins-hbase4.apache.org,42517,1679838814721, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:53:52,606 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/cf49ed7226a84ca495773eebae7b2c4e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/cf49ed7226a84ca495773eebae7b2c4e 2023-03-26 13:53:52,614 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in c54bfce2a8106a499411906befe41bd8/A of c54bfce2a8106a499411906befe41bd8 into cf49ed7226a84ca495773eebae7b2c4e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:52,614 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:52,615 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/A, priority=12, startTime=1679838832157; duration=0sec 2023-03-26 13:53:52,615 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:52,645 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/fc63ed77dfd3435893df50d979cb24a0 2023-03-26 13:53:52,652 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/f1cdbb3327724c7381eb39c4a4c3cd8d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/f1cdbb3327724c7381eb39c4a4c3cd8d 2023-03-26 13:53:52,659 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/C of c54bfce2a8106a499411906befe41bd8 into f1cdbb3327724c7381eb39c4a4c3cd8d(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:52,659 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:52,659 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/C, priority=13, startTime=1679838832158; duration=0sec 2023-03-26 13:53:52,660 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:52,673 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=477 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/e13b16c12cda44f89f568cab5c333299 2023-03-26 13:53:52,679 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/1fa60de424bd434caabbed8657aec23d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1fa60de424bd434caabbed8657aec23d 2023-03-26 13:53:52,684 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1fa60de424bd434caabbed8657aec23d, entries=150, sequenceid=477, filesize=11.9 K 2023-03-26 13:53:52,685 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/fc63ed77dfd3435893df50d979cb24a0 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/fc63ed77dfd3435893df50d979cb24a0 2023-03-26 13:53:52,690 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/fc63ed77dfd3435893df50d979cb24a0, entries=150, sequenceid=477, filesize=11.9 K 2023-03-26 13:53:52,690 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/e13b16c12cda44f89f568cab5c333299 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e13b16c12cda44f89f568cab5c333299 2023-03-26 13:53:52,697 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e13b16c12cda44f89f568cab5c333299, entries=150, sequenceid=477, filesize=11.9 K 2023-03-26 13:53:52,698 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~345.09 KB/353376, currentSize=60.38 KB/61830 for c54bfce2a8106a499411906befe41bd8 in 516ms, sequenceid=477, compaction requested=false 2023-03-26 13:53:52,698 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:52,698 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:52,698 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:52,699 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:52,699 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,699 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:52,699 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:52,700 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,700 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:52,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:52,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:52,701 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,701 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:52,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:52,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:52,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23db1b64[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:52,703 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:52,703 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,706 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,706 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:52,706 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:52,706 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:52,708 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:52,708 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:52,708 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:52,709 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,709 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:52,709 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:52,709 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:52,709 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:52,709 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:52,709 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:52,709 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:52,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:52,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:52,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:52,712 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,714 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:52,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:52,715 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-26 13:53:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:52,715 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:52,715 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:52,715 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:52,716 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:52,716 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:52,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,717 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:52,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:52,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:52,718 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:52,718 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:52,718 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:52,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:52,718 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:52,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:52,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:52,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:52,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838892744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:52,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838892745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,818 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:52,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:52,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:52,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:52,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:52,822 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:52,822 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:52,822 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:52,822 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:52,822 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,822 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,824 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:52,824 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:52,824 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,824 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:52,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,825 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:52,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:52,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:52,826 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:52,826 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:52,826 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:52,828 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:52,828 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:52,828 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:52,828 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:52,828 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:52,828 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:52,828 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:52,829 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:52,829 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,829 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:52,829 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:52,831 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:52,831 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:52,832 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,832 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:52,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:52,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:52,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62386d1a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:52,834 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:52,834 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,835 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,835 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:52,835 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:52,835 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:52,838 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:52,840 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,841 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:52,841 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:52,841 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:52,841 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:52,841 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:52,841 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:52,841 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:52,843 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:52,843 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,843 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:52,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:52,844 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:52,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:52,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:52,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:52,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838892847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:52,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838892850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,852 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:52,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:52,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:52,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:52,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:52,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:52,857 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:52,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:52,857 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:52,858 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:52,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:52,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:52,958 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:52,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:52,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:52,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:52,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:52,961 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:52,961 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:52,961 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:52,961 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:52,962 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,962 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,963 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:52,963 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:52,964 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,964 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:52,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:52,965 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:52,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:52,966 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:52,966 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:52,966 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:52,966 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:52,966 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:52,966 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:52,966 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:52,966 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:52,966 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,966 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:52,966 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:52,968 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,968 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:52,968 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,968 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:52,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:52,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69c74c7c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:52,970 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:52,970 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,971 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,971 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:52,971 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,971 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:52,971 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:52,971 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:52,974 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,974 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:52,974 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:52,974 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:52,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:52,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:52,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:52,976 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:52,976 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:52,976 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:52,977 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:52,977 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:52,978 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,978 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:52,978 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:52,979 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:52,981 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:52,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:52,985 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,985 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,985 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:52,985 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:52,985 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,985 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:52,985 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:52,985 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:52,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:52,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:52,985 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:52,985 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:52,985 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:52,985 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:52,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:52,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:52,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:52,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:53,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:53,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838893052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:53,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838893052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,086 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:53,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:53,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:53,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:53,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:53,089 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:53,089 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:53,089 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:53,089 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:53,090 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,090 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,095 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:53,095 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:53,095 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,095 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:53,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:53,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:53,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:53,097 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:53,097 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:53,097 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:53,108 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:53,108 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:53,108 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:53,108 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:53,108 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:53,108 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:53,108 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:53,108 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:53,108 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,108 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:53,108 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:53,110 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,110 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:53,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:53,110 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,110 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:53,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:53,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16fb0a0e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:53,112 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:53,112 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,115 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,115 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:53,115 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,115 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:53,115 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:53,115 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:53,117 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:53,117 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,117 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:53,117 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:53,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:53,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:53,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:53,120 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:53,120 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:53,120 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:53,120 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:53,120 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:53,122 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,122 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:53,122 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:53,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:53,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:53,123 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:53,124 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,127 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,127 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,128 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:53,128 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,128 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:53,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:53,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:53,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:53,128 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:53,128 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:53,128 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,128 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:53,128 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:53,128 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,128 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:53,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:53,132 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/62f0c4eca80c41f7b4b9b5fdc945e6f6 2023-03-26 13:53:53,147 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/f5e24985863943baa7020623cdd85437 2023-03-26 13:53:53,228 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:53,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:53,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:53,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:53,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:53,235 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:53,235 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:53,235 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:53,235 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:53,235 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,235 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,237 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:53,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:53,237 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,237 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,237 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:53,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:53,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:53,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:53,238 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:53,239 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:53,239 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:53,239 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:53,239 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:53,239 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:53,239 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:53,239 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:53,239 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:53,239 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:53,239 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:53,239 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,239 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:53,239 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:53,241 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,241 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:53,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:53,241 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,241 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:53,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:53,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:53,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:53,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d80238e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:53,245 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:53,245 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,246 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,246 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,246 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:53,246 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:53,246 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:53,246 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:53,250 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:53,250 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:53,250 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,250 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:53,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:53,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:53,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:53,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:53,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:53,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:53,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:53,253 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:53,253 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:53,253 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:53,253 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:53,253 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:53,257 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,257 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:53,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,257 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:53,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:53,258 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,259 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,261 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,261 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,261 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,261 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,261 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:53,261 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:53,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:53,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:53,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:53,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:53,261 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,262 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,261 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:53,262 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:53,262 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:53,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:53,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:53,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838893356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:53,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838893357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,362 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:53,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:53,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:53,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:53,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:53,365 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:53,365 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:53,366 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:53,366 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:53,366 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,366 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,368 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,368 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:53,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:53,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:53,368 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,369 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:53,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:53,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:53,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:53,370 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:53,370 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:53,370 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:53,370 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:53,370 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:53,370 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:53,370 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:53,370 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:53,370 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:53,371 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:53,371 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:53,371 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,371 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:53,371 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:53,372 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,372 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:53,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:53,373 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,373 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:53,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:53,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:53,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c71d399[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:53,374 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:53,374 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,375 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:53,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:53,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:53,376 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,376 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:53,377 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:53,377 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:53,377 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,377 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:53,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:53,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:53,379 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:53,380 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:53,380 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:53,380 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:53,380 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:53,382 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,382 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,382 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:53,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:53,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,383 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:53,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:53,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:53,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,384 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:53,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:53,387 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,387 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:53,387 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:53,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:53,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:53,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:53,387 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,387 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:53,387 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:53,387 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,387 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,387 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:53,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:53,387 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:53,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:53,387 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:53,387 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:53,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838893440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:53,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838893442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,487 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:53,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:53,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:53,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:53,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:53,493 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:53,493 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:53,493 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:53,493 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:53,494 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,494 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,500 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:53,500 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:53,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:53,501 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,501 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:53,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:53,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:53,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:53,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:53,508 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:53,508 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:53,508 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:53,515 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:53,515 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:53,515 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:53,515 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:53,515 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:53,515 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:53,515 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:53,515 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:53,515 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,515 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:53,515 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:53,517 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,517 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:53,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:53,517 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,517 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:53,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:53,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:53,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:53,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:53,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f06d2c7[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:53,519 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:53,519 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,524 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:53,524 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:53,524 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,524 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:53,525 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,525 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:53,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:53,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:53,525 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:53,526 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:53,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:53,528 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:53,528 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:53,528 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:53,528 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:53,528 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:53,530 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,530 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:53,530 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:53,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:53,531 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,532 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:53,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:53,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:53,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:53,535 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:53,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:53,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:53,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:53,536 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:53,536 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:53,536 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:53,570 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=493 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/237bfb7f6e41499982d7be263bb588f9 2023-03-26 13:53:53,576 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/62f0c4eca80c41f7b4b9b5fdc945e6f6 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/62f0c4eca80c41f7b4b9b5fdc945e6f6 2023-03-26 13:53:53,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/62f0c4eca80c41f7b4b9b5fdc945e6f6, entries=200, sequenceid=493, filesize=14.3 K 2023-03-26 13:53:53,583 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/f5e24985863943baa7020623cdd85437 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/f5e24985863943baa7020623cdd85437 2023-03-26 13:53:53,591 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/f5e24985863943baa7020623cdd85437, entries=150, sequenceid=493, filesize=11.9 K 2023-03-26 13:53:53,592 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/237bfb7f6e41499982d7be263bb588f9 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/237bfb7f6e41499982d7be263bb588f9 2023-03-26 13:53:53,597 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/237bfb7f6e41499982d7be263bb588f9, entries=150, sequenceid=493, filesize=11.9 K 2023-03-26 13:53:53,598 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for c54bfce2a8106a499411906befe41bd8 in 883ms, sequenceid=493, compaction requested=true 2023-03-26 13:53:53,598 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:53,599 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:53,599 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:53,599 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:53,599 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:53,599 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:53,601 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:53,601 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37614 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:53,601 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/A is initiating minor compaction (all files) 2023-03-26 13:53:53,601 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/B is initiating minor compaction (all files) 2023-03-26 13:53:53,601 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/A in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:53,601 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/B in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:53,601 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/cf49ed7226a84ca495773eebae7b2c4e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1fa60de424bd434caabbed8657aec23d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/62f0c4eca80c41f7b4b9b5fdc945e6f6] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=39.1 K 2023-03-26 13:53:53,601 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/3cd5368930bf447b83b4e0856113ac15, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/fc63ed77dfd3435893df50d979cb24a0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/f5e24985863943baa7020623cdd85437] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=36.7 K 2023-03-26 13:53:53,602 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting cf49ed7226a84ca495773eebae7b2c4e, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1679838831209 2023-03-26 13:53:53,602 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 3cd5368930bf447b83b4e0856113ac15, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1679838831209 2023-03-26 13:53:53,602 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 1fa60de424bd434caabbed8657aec23d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1679838831290 2023-03-26 13:53:53,602 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting fc63ed77dfd3435893df50d979cb24a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1679838831290 2023-03-26 13:53:53,603 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 62f0c4eca80c41f7b4b9b5fdc945e6f6, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1679838832402 2023-03-26 13:53:53,603 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting f5e24985863943baa7020623cdd85437, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1679838832404 2023-03-26 13:53:53,620 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#A#compaction#99 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:53,620 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#B#compaction#100 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:53,640 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:53,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:53,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:53,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:53,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:53,644 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:53,644 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:53,644 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:53,644 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:53,645 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,645 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,647 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:53,647 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:53,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:53,647 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,647 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:53,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:53,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:53,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:53,654 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:53,654 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:53,654 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:53,654 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:53,654 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:53,654 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:53,654 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:53,654 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-26 13:53:53,655 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:53,655 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:53,655 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:53,655 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:53,655 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:53,655 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:53,669 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/4c017c3cc0f24ea4af4b049966904ed7 2023-03-26 13:53:53,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:53,863 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:53,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:53,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838893880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:53,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838893881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:53,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838893984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:53,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:53,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838893986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,041 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/f6cf98454d98418d92b10b950fa31df1 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/f6cf98454d98418d92b10b950fa31df1 2023-03-26 13:53:54,042 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/29d98a4d16cc41b38d947afc1f1b3958 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/29d98a4d16cc41b38d947afc1f1b3958 2023-03-26 13:53:54,049 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/B of c54bfce2a8106a499411906befe41bd8 into f6cf98454d98418d92b10b950fa31df1(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:54,049 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:54,049 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/B, priority=13, startTime=1679838833599; duration=0sec 2023-03-26 13:53:54,049 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:54,049 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:54,051 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:54,051 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/A of c54bfce2a8106a499411906befe41bd8 into 29d98a4d16cc41b38d947afc1f1b3958(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:54,051 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/C is initiating minor compaction (all files) 2023-03-26 13:53:54,051 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:54,051 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/C in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:54,051 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/A, priority=13, startTime=1679838833599; duration=0sec 2023-03-26 13:53:54,051 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/f1cdbb3327724c7381eb39c4a4c3cd8d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e13b16c12cda44f89f568cab5c333299, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/237bfb7f6e41499982d7be263bb588f9] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=36.7 K 2023-03-26 13:53:54,051 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:54,052 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting f1cdbb3327724c7381eb39c4a4c3cd8d, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=442, earliestPutTs=1679838831209 2023-03-26 13:53:54,053 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e13b16c12cda44f89f568cab5c333299, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=477, earliestPutTs=1679838831290 2023-03-26 13:53:54,053 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 237bfb7f6e41499982d7be263bb588f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1679838832404 2023-03-26 13:53:54,066 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#C#compaction#103 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:54,086 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/e175e428702c4d58b75b4f204631ed19 2023-03-26 13:53:54,099 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/ebf78b54ecfb4e1498bd3cf5b04b1134 2023-03-26 13:53:54,104 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/4c017c3cc0f24ea4af4b049966904ed7 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/4c017c3cc0f24ea4af4b049966904ed7 2023-03-26 13:53:54,109 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/4c017c3cc0f24ea4af4b049966904ed7, entries=150, sequenceid=512, filesize=11.9 K 2023-03-26 13:53:54,110 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/e175e428702c4d58b75b4f204631ed19 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e175e428702c4d58b75b4f204631ed19 2023-03-26 13:53:54,115 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e175e428702c4d58b75b4f204631ed19, entries=150, sequenceid=512, filesize=11.9 K 2023-03-26 13:53:54,116 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/ebf78b54ecfb4e1498bd3cf5b04b1134 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/ebf78b54ecfb4e1498bd3cf5b04b1134 2023-03-26 13:53:54,121 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/ebf78b54ecfb4e1498bd3cf5b04b1134, entries=150, sequenceid=512, filesize=11.9 K 2023-03-26 13:53:54,122 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for c54bfce2a8106a499411906befe41bd8 in 468ms, sequenceid=512, compaction requested=false 2023-03-26 13:53:54,122 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:54,122 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:54,122 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:54,122 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:54,122 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,122 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:54,122 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:54,123 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,123 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,124 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,124 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:54,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51e31053[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:54,126 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:54,126 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,128 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,128 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:54,128 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:54,128 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:54,128 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,128 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:54,130 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,130 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:54,130 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,130 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:54,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:54,133 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:54,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:54,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:54,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:54,133 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:54,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:54,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,136 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,136 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,139 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,141 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,141 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,141 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,141 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,141 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:54,141 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:54,141 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:54,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:54,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:54,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:54,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:54,141 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,142 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,142 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,142 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:54,142 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:54,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:54,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:54,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:54,187 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-26 13:53:54,188 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:54,188 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:54,188 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:54,188 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:54,188 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:54,188 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:54,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:54,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838894211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:54,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838894212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,242 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:54,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:54,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:54,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:54,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:54,252 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:54,252 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:54,252 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:54,252 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:54,253 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,253 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,254 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:54,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:54,254 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:54,254 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,254 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:54,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,255 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:54,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:54,256 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:54,256 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:54,256 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:54,257 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:54,257 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:54,257 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:54,257 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:54,257 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:54,257 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:54,257 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:54,257 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:54,257 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,257 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:54,257 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:54,261 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,261 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,261 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,261 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:54,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:54,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d93c5b2[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:54,263 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:54,263 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,265 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:54,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:54,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:54,265 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,265 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:54,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:54,268 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:54,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:54,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:54,271 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:54,271 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:54,271 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:54,271 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:54,271 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:54,272 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,272 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,272 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:54,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:54,273 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,274 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:54,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:54,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:54,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:54,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:54,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:54,276 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:54,276 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:54,276 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:54,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:54,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:54,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:54,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:54,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838894315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,316 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:54,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838894315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,376 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:54,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:54,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:54,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:54,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:54,379 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:54,380 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:54,380 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:54,380 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:54,380 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,380 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,381 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,381 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:54,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:54,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:54,382 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,382 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:54,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:54,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:54,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:54,383 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:54,383 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:54,383 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:54,383 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:54,383 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:54,383 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:54,384 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:54,384 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:54,384 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:54,384 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:54,384 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:54,384 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,384 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:54,384 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:54,387 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,387 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:54,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fbc3482[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:54,389 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:54,390 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,391 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,391 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:54,391 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:54,391 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:54,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:54,395 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,395 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:54,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,395 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:54,395 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:54,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:54,398 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:54,398 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:54,398 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:54,398 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:54,398 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:54,400 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,400 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:54,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,400 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:54,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,402 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,404 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:54,404 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:54,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:54,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:54,404 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:54,404 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:54,404 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:54,404 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:54,404 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,404 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,404 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,404 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,404 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,404 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:54,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:54,485 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/09be9cf40b0542718c950c25e3ba69cd as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/09be9cf40b0542718c950c25e3ba69cd 2023-03-26 13:53:54,492 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/C of c54bfce2a8106a499411906befe41bd8 into 09be9cf40b0542718c950c25e3ba69cd(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:54,492 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:54,492 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/C, priority=13, startTime=1679838833599; duration=0sec 2023-03-26 13:53:54,492 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:54,504 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:54,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:54,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:54,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:54,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:54,512 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:54,512 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:54,512 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:54,512 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:54,513 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,513 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,515 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:54,515 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:54,516 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,516 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:54,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:54,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:54,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:54,517 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:54,517 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:54,517 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:54,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:54,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838894517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:54,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838894518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,520 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:54,520 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:54,520 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:54,520 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:54,520 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:54,520 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:54,520 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:54,520 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:54,520 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,520 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:54,520 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:54,522 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,522 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,523 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,523 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:54,524 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:54,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56855158[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:54,524 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,525 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,526 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:54,526 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,526 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:54,526 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:54,526 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:54,527 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,528 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:54,528 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:54,528 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:54,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:54,530 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:54,531 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:54,531 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:54,531 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:54,531 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:54,533 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,533 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,533 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:54,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:54,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:54,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,534 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,537 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,539 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,540 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:54,540 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:54,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:54,540 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:54,540 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,540 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,540 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,540 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,540 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:54,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:54,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:54,540 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:54,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:54,540 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:54,540 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:54,540 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:54,600 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/8ec9a13eabf34a21a1747a7a78dc0a24 2023-03-26 13:53:54,641 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:54,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:54,642 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/4374697ddaa246a1a3aa049fc48b0838 2023-03-26 13:53:54,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:54,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:54,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:54,644 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:54,645 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:54,645 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:54,645 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:54,645 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,645 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,646 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:54,646 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,647 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,647 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:54,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:54,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:54,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:54,652 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:54,668 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:54,668 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:54,680 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:54,680 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:54,680 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:54,680 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,691 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:54,691 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:54,695 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,696 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,696 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:54,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f2db253[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:54,698 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:54,698 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,699 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,699 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:54,699 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,700 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:54,700 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:54,700 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:54,704 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,705 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=528 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/e6d50a4d0d364e95a9c1dd6bb762bb61 2023-03-26 13:53:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:54,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:54,707 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:54,707 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:54,707 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:54,707 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:54,707 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:54,709 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,709 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,710 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,711 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,715 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,719 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/8ec9a13eabf34a21a1747a7a78dc0a24 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/8ec9a13eabf34a21a1747a7a78dc0a24 2023-03-26 13:53:54,720 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,720 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,720 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,720 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:54,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:54,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:54,720 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,720 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,720 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,720 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,721 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:54,721 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:54,722 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:54,722 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:54,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:54,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:54,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/8ec9a13eabf34a21a1747a7a78dc0a24, entries=200, sequenceid=528, filesize=14.3 K 2023-03-26 13:53:54,729 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/4374697ddaa246a1a3aa049fc48b0838 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/4374697ddaa246a1a3aa049fc48b0838 2023-03-26 13:53:54,742 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/4374697ddaa246a1a3aa049fc48b0838, entries=150, sequenceid=528, filesize=11.9 K 2023-03-26 13:53:54,743 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/e6d50a4d0d364e95a9c1dd6bb762bb61 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e6d50a4d0d364e95a9c1dd6bb762bb61 2023-03-26 13:53:54,750 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e6d50a4d0d364e95a9c1dd6bb762bb61, entries=150, sequenceid=528, filesize=11.9 K 2023-03-26 13:53:54,751 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c54bfce2a8106a499411906befe41bd8 in 564ms, sequenceid=528, compaction requested=true 2023-03-26 13:53:54,751 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:54,752 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:54,752 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:54,752 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:54,752 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:54,752 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:54,753 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:54,753 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:54,753 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/A is initiating minor compaction (all files) 2023-03-26 13:53:54,753 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/B is initiating minor compaction (all files) 2023-03-26 13:53:54,753 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/A in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:54,753 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/B in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:54,754 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/29d98a4d16cc41b38d947afc1f1b3958, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/4c017c3cc0f24ea4af4b049966904ed7, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/8ec9a13eabf34a21a1747a7a78dc0a24] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=39.2 K 2023-03-26 13:53:54,754 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/f6cf98454d98418d92b10b950fa31df1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e175e428702c4d58b75b4f204631ed19, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/4374697ddaa246a1a3aa049fc48b0838] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=36.8 K 2023-03-26 13:53:54,755 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 29d98a4d16cc41b38d947afc1f1b3958, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1679838832404 2023-03-26 13:53:54,755 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting f6cf98454d98418d92b10b950fa31df1, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1679838832404 2023-03-26 13:53:54,756 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 4c017c3cc0f24ea4af4b049966904ed7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1679838832731 2023-03-26 13:53:54,756 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e175e428702c4d58b75b4f204631ed19, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1679838832731 2023-03-26 13:53:54,756 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 8ec9a13eabf34a21a1747a7a78dc0a24, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1679838833873 2023-03-26 13:53:54,757 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 4374697ddaa246a1a3aa049fc48b0838, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1679838833879 2023-03-26 13:53:54,773 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#A#compaction#108 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:54,781 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#B#compaction#109 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:54,817 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/1ad273f134e349c49539a1496feb965a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1ad273f134e349c49539a1496feb965a 2023-03-26 13:53:54,821 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:54,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:54,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:54,824 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-26 13:53:54,825 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/A of c54bfce2a8106a499411906befe41bd8 into 1ad273f134e349c49539a1496feb965a(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:54,825 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:54,825 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/A, priority=13, startTime=1679838834752; duration=0sec 2023-03-26 13:53:54,825 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:54,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:54,825 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:53:54,827 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:53:54,827 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/C is initiating minor compaction (all files) 2023-03-26 13:53:54,827 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/C in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:54,827 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:54,827 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/09be9cf40b0542718c950c25e3ba69cd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/ebf78b54ecfb4e1498bd3cf5b04b1134, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e6d50a4d0d364e95a9c1dd6bb762bb61] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=36.8 K 2023-03-26 13:53:54,827 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:54,827 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:54,827 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:54,827 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:54,827 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:54,827 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 09be9cf40b0542718c950c25e3ba69cd, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=493, earliestPutTs=1679838832404 2023-03-26 13:53:54,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:54,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:54,828 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:54,828 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:54,828 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:54,828 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:54,828 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting ebf78b54ecfb4e1498bd3cf5b04b1134, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1679838832731 2023-03-26 13:53:54,829 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting e6d50a4d0d364e95a9c1dd6bb762bb61, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1679838833879 2023-03-26 13:53:54,829 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,829 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,831 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:54,831 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:54,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:54,831 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,831 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:54,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:54,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:54,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:54,837 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:54,837 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:54,837 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:54,838 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:54,838 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:54,838 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:54,838 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:54,838 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:54,839 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:54,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:54,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:54,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,839 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#C#compaction#111 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:54,840 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:54,840 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:54,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,842 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,843 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,843 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,844 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=550 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/1cabde85fd1547fb93f275c581dff7fa 2023-03-26 13:53:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:54,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c304a51[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:54,846 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:54,846 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,848 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,848 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:54,848 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:54,848 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:54,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:54,850 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:54,850 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:54,851 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:54,851 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:54,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838894857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:54,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:54,860 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:54,860 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=550 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/896f0656f1594333b101e9a24a3ae17a 2023-03-26 13:53:54,860 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/da98c6bf700641a28dda7c8fa6405326 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/da98c6bf700641a28dda7c8fa6405326 2023-03-26 13:53:54,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838894857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,860 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:54,861 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:54,861 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:54,861 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:54,861 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:54,863 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,863 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,863 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,864 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,868 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,869 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/C of c54bfce2a8106a499411906befe41bd8 into da98c6bf700641a28dda7c8fa6405326(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:54,869 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:54,870 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/C, priority=13, startTime=1679838834752; duration=0sec 2023-03-26 13:53:54,870 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:54,875 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:54,875 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:54,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:54,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:54,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:54,875 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:54,876 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:54,876 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:54,876 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,876 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,876 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,876 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,876 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:54,876 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:54,876 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:54,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838894962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,963 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:54,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838894962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,976 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:54,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:54,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:54,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:54,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:54,979 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:54,979 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:54,979 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:54,979 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:54,980 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,980 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,981 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,981 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:54,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:54,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:54,982 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,982 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:54,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:54,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:54,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:54,983 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:54,983 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:54,983 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:54,990 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:54,990 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:54,990 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:54,990 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:54,990 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:54,990 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:54,991 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:54,992 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,992 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,993 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:54,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22367171[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:54,995 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:54,995 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,996 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:54,997 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:54,997 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:54,997 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,997 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:54,997 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:54,998 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:55,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:55,003 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:55,003 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:55,003 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:55,003 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:55,003 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:55,021 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,022 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,022 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:55,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:55,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:55,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:55,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:55,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:55,022 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:55,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:55,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,024 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:55,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:55,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,027 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,027 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:55,027 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:55,027 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:55,027 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:55,027 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:55,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:55,027 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:55,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:55,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:55,027 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:55,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:55,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:55,028 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:55,028 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:55,028 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,028 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:55,128 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:55,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:55,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:55,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:55,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:55,134 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:55,134 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:55,134 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:55,134 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:55,135 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,135 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:55,136 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,136 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:55,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:55,137 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,137 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:55,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:55,140 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:55,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:55,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:55,140 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:55,140 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:55,140 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:55,140 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:55,140 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:55,140 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:55,140 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:55,140 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:55,141 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:55,141 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:55,141 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:55,141 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,141 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:55,141 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:55,143 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:55,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:55,144 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:55,144 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:55,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:55,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e45ebfe[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:55,145 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:55,145 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:55,146 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:55,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:55,146 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:55,146 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:55,146 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,146 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:55,147 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:55,148 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:55,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:55,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:55,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:55,150 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:55,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:55,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:55,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:55,150 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:55,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:55,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:55,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:55,152 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:55,154 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:55,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,157 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,157 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:55,157 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:55,158 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:55,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:55,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:55,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:55,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:55,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:55,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:55,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:55,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:55,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:55,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,159 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:55,159 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:55,168 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:55,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838895164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:55,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838895165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,221 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/8c580fea9dea44fcaf8dd033d64a6b09 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/8c580fea9dea44fcaf8dd033d64a6b09 2023-03-26 13:53:55,226 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c54bfce2a8106a499411906befe41bd8/B of c54bfce2a8106a499411906befe41bd8 into 8c580fea9dea44fcaf8dd033d64a6b09(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:55,226 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:55,226 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/B, priority=13, startTime=1679838834752; duration=0sec 2023-03-26 13:53:55,226 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:55,259 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:55,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:55,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:55,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:55,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:55,262 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:55,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:55,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:55,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:55,263 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,263 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:55,264 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:55,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:55,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:55,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:55,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:55,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:55,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:55,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:55,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:55,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:55,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:55,266 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:55,266 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:55,266 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:55,266 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:55,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:55,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:55,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:55,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:55,268 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:55,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:55,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:55,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:55,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:55,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c08754f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:55,270 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:55,270 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:55,271 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:55,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:55,271 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:55,271 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:55,271 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:55,271 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,271 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:55,274 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,274 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:55,274 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:55,274 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:55,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:55,276 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:55,276 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:55,277 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:55,277 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:55,277 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:55,278 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,278 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:55,278 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:55,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:55,279 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:55,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:55,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,285 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,285 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:55,285 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:55,285 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:55,285 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:55,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:55,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:55,285 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:55,285 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:55,286 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:55,286 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,286 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:55,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:55,286 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:55,286 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:55,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:55,315 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=550 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/8ee877fc8ff44cc8821deffed71a4790 2023-03-26 13:53:55,324 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/1cabde85fd1547fb93f275c581dff7fa as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1cabde85fd1547fb93f275c581dff7fa 2023-03-26 13:53:55,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1cabde85fd1547fb93f275c581dff7fa, entries=150, sequenceid=550, filesize=11.9 K 2023-03-26 13:53:55,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/896f0656f1594333b101e9a24a3ae17a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/896f0656f1594333b101e9a24a3ae17a 2023-03-26 13:53:55,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/896f0656f1594333b101e9a24a3ae17a, entries=150, sequenceid=550, filesize=11.9 K 2023-03-26 13:53:55,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/8ee877fc8ff44cc8821deffed71a4790 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/8ee877fc8ff44cc8821deffed71a4790 2023-03-26 13:53:55,344 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/8ee877fc8ff44cc8821deffed71a4790, entries=150, sequenceid=550, filesize=11.9 K 2023-03-26 13:53:55,345 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for c54bfce2a8106a499411906befe41bd8 in 521ms, sequenceid=550, compaction requested=false 2023-03-26 13:53:55,345 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:55,386 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:55,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:55,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:55,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:55,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:55,390 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:55,390 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:55,390 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:55,390 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:55,390 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,390 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:55,394 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:55,394 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:55,395 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,395 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:55,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:55,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:55,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:55,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:55,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:55,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:55,410 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:55,410 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:55,410 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:55,410 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:55,410 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-26 13:53:55,410 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:55,411 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:55,411 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:55,411 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:55,411 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:55,411 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:55,420 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/2a317b7b71774ea091e236198ac9edf5 2023-03-26 13:53:55,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:55,455 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:55,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:55,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838895471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:55,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838895471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:55,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838895471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:55,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838895472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:55,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:55,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838895575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838895575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,578 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:55,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:55,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838895576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838895575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:55,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838895779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:55,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838895779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:55,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838895779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:55,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838895779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:55,832 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/f2a7fc5080a34902a5ec19455c5862bc 2023-03-26 13:53:56,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:56,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838896082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,083 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:56,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838896082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:56,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838896083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:56,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838896083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,246 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/022d0627cde54e8f9eb3d097d7b23c35 2023-03-26 13:53:56,251 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/2a317b7b71774ea091e236198ac9edf5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/2a317b7b71774ea091e236198ac9edf5 2023-03-26 13:53:56,256 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/2a317b7b71774ea091e236198ac9edf5, entries=150, sequenceid=565, filesize=11.9 K 2023-03-26 13:53:56,257 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/f2a7fc5080a34902a5ec19455c5862bc as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/f2a7fc5080a34902a5ec19455c5862bc 2023-03-26 13:53:56,266 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/f2a7fc5080a34902a5ec19455c5862bc, entries=150, sequenceid=565, filesize=11.9 K 2023-03-26 13:53:56,267 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/022d0627cde54e8f9eb3d097d7b23c35 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/022d0627cde54e8f9eb3d097d7b23c35 2023-03-26 13:53:56,271 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/022d0627cde54e8f9eb3d097d7b23c35, entries=150, sequenceid=565, filesize=11.9 K 2023-03-26 13:53:56,272 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for c54bfce2a8106a499411906befe41bd8 in 862ms, sequenceid=565, compaction requested=true 2023-03-26 13:53:56,272 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:56,272 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:56,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:56,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:56,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:56,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:56,277 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:56,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:56,278 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:56,278 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:56,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:56,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:56,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12d4c05d[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:56,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:56,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:56,282 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:56,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:56,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:56,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:56,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:56,282 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,282 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:56,284 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:56,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:56,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:56,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:56,287 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:56,287 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:56,287 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:56,287 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:56,287 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:56,289 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:56,289 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:56,289 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:56,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:56,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:56,290 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:56,292 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:56,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:56,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:56,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:56,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:56,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:56,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:56,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:56,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:56,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:56,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:56,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:56,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:56,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:56,295 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:56,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:56,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:56,296 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:56,296 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:56,396 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:56,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:56,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:56,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:56,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:56,400 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:56,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:56,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:56,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:56,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:56,401 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:56,402 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:56,402 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:56,403 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,403 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:56,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:56,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:56,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:56,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:56,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:56,405 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:56,406 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:56,406 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:56,406 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:56,406 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-26 13:53:56,406 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:56,406 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:56,407 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:56,407 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:56,407 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:56,407 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:56,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:56,588 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:56,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:56,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838896600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:56,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838896603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:56,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838896603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:56,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838896605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,608 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:56,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39044 deadline: 1679838896606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,610 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., hostname=jenkins-hbase4.apache.org,42517,1679838814721, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:53:56,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:56,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838896706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:56,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838896706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:56,708 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:56,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838896706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838896707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,819 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/b3d2c2e96d5d45e3a5c288529c7d913a 2023-03-26 13:53:56,831 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/e12c5fbf12b345e09ac552910b5b3745 2023-03-26 13:53:56,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:56,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:56,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838896910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838896910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:56,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838896910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:56,911 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:56,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838896911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:57,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838897211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:57,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838897213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:57,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838897213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:57,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838897214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,243 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=584 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/39ad0277b2c8424fb05f4f464a001a0e 2023-03-26 13:53:57,249 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/b3d2c2e96d5d45e3a5c288529c7d913a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/b3d2c2e96d5d45e3a5c288529c7d913a 2023-03-26 13:53:57,254 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/b3d2c2e96d5d45e3a5c288529c7d913a, entries=150, sequenceid=584, filesize=11.9 K 2023-03-26 13:53:57,255 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/e12c5fbf12b345e09ac552910b5b3745 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e12c5fbf12b345e09ac552910b5b3745 2023-03-26 13:53:57,259 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e12c5fbf12b345e09ac552910b5b3745, entries=150, sequenceid=584, filesize=11.9 K 2023-03-26 13:53:57,260 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/39ad0277b2c8424fb05f4f464a001a0e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/39ad0277b2c8424fb05f4f464a001a0e 2023-03-26 13:53:57,267 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/39ad0277b2c8424fb05f4f464a001a0e, entries=150, sequenceid=584, filesize=11.9 K 2023-03-26 13:53:57,268 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for c54bfce2a8106a499411906befe41bd8 in 862ms, sequenceid=584, compaction requested=true 2023-03-26 13:53:57,268 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:57,268 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:57,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:57,270 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,270 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:57,270 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,270 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:57,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:57,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:57,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:57,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:57,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f8868f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:57,272 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:57,272 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,274 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,274 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:57,274 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:57,274 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:57,274 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,274 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:57,275 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:57,275 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,275 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:57,275 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:57,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:57,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:57,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:57,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:57,279 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:57,279 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:57,279 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:57,279 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:57,279 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:57,282 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,282 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,282 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:57,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:57,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,283 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,284 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:57,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,286 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,287 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,287 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,287 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,287 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,287 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:57,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:57,287 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:57,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:57,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:57,287 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,287 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:57,287 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:57,287 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:57,287 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:57,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:57,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:57,387 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:57,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:57,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:57,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:57,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:57,391 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:57,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:57,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:57,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:57,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,392 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,399 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,400 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:57,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:57,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:57,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:57,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:57,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:57,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:57,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:57,401 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:57,401 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:57,401 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:57,401 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:57,401 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:57,401 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:57,402 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:57,402 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-26 13:53:57,402 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:57,402 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:57,402 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:57,402 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:57,402 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:57,402 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:57,413 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=598 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/d531fa34f632475996ab547785186a6e 2023-03-26 13:53:57,435 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=598 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/22876eb85bfa4900823f06c41d6cf471 2023-03-26 13:53:57,461 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=598 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/73a4a240178f4c21ac4038e9922ff4bc 2023-03-26 13:53:57,467 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/d531fa34f632475996ab547785186a6e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/d531fa34f632475996ab547785186a6e 2023-03-26 13:53:57,474 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/d531fa34f632475996ab547785186a6e, entries=150, sequenceid=598, filesize=11.9 K 2023-03-26 13:53:57,475 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/22876eb85bfa4900823f06c41d6cf471 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/22876eb85bfa4900823f06c41d6cf471 2023-03-26 13:53:57,480 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/22876eb85bfa4900823f06c41d6cf471, entries=150, sequenceid=598, filesize=11.9 K 2023-03-26 13:53:57,481 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/73a4a240178f4c21ac4038e9922ff4bc as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/73a4a240178f4c21ac4038e9922ff4bc 2023-03-26 13:53:57,489 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/73a4a240178f4c21ac4038e9922ff4bc, entries=150, sequenceid=598, filesize=11.9 K 2023-03-26 13:53:57,490 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for c54bfce2a8106a499411906befe41bd8 in 88ms, sequenceid=598, compaction requested=true 2023-03-26 13:53:57,490 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:57,490 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:57,490 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:57,491 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:57,491 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,492 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:57,492 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:57,494 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:57,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,494 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:57,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:57,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:57,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72939715[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:57,499 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:57,499 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,501 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,501 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:57,501 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:57,501 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:57,502 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,502 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:57,503 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:57,503 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:57,503 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:57,503 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:57,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:57,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:57,506 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:57,506 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:57,506 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:57,506 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:57,506 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:57,509 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:57,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:57,509 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,510 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:57,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:57,510 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:57,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,514 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:57,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,521 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,521 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,521 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:57,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:57,521 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:57,521 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:57,521 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,521 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,521 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:57,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:57,521 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,521 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,521 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:57,522 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:57,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:57,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:57,622 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:57,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:57,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:57,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:57,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:57,627 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:57,627 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:57,627 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:57,627 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:57,628 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,628 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,631 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:57,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:57,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:57,631 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,632 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,632 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:57,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:57,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:57,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:57,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:57,643 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:57,644 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:57,644 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:57,644 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:57,644 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:57,644 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:57,644 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:57,644 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:57,644 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:57,644 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:57,644 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:57,644 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,644 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:57,645 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:57,646 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,646 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:57,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:57,647 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,647 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:57,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:57,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:57,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dde2605[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:57,650 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:57,650 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,651 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,651 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,651 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:57,651 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:57,651 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:57,652 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:57,654 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,654 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:57,655 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:57,655 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:57,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:57,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:57,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:57,661 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:57,661 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:57,661 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:57,661 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:57,661 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:57,664 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,664 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,664 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:57,665 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:57,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,667 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:57,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,672 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:57,672 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:57,672 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,672 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,672 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,672 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,672 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,672 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,672 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:57,672 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:57,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:57,673 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:57,673 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:57,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:57,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:57,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:57,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:57,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on c54bfce2a8106a499411906befe41bd8 2023-03-26 13:53:57,728 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:53:57,728 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:57,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:57,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:57,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:57,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:57,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:53:57,773 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:57,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:57,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:57,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:57,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:57,784 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:57,784 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:57,784 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:57,784 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:57,784 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,784 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838837769, max timestamp=1679838837783Num uniques 859; 2023-03-26 13:53:57,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:53:57,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838837769, max timestamp=1679838837783Num uniques 859; 2023-03-26 13:53:57,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838837769, max timestamp=1679838837783Num uniques 859; 2023-03-26 13:53:57,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:53:57,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:53:57,786 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,786 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:57,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,787 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,787 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:57,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:57,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:57,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:57,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:57,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:57,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:57,803 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:57,803 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:57,803 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:57,803 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:57,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:57,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:57,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:57,804 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:57,805 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,805 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,806 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:57,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:57,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:57,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29bcdbcb[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:57,810 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:57,810 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,815 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:57,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:57,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:57,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:57,817 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:57,817 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,817 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:57,817 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:57,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:57,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:57,821 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:57,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:57,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:57,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:57,821 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:57,823 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,823 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:57,823 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:57,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,824 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,824 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:57,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838897810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:57,826 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:57,831 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:57,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:57,831 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:57,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838897821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,831 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,832 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,832 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,832 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,832 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,832 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,831 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:57,832 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:57,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:57,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:57,831 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:57,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838897823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,832 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:57,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:57,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:57,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838897825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:57,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838897926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,932 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:57,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:57,935 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:57,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:57,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838897933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:57,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:57,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:57,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:57,937 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:57,937 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:57,937 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:57,937 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:57,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838897934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838897935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,937 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,937 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:57,940 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:57,940 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,940 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:57,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:57,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,941 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:57,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:57,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:57,942 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:57,942 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:57,942 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:57,944 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:57,944 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:57,944 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:57,944 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:57,944 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:57,944 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:57,944 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:57,944 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:57,944 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,944 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:57,944 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:57,946 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,946 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:57,946 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,946 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:57,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:57,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:57,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e37fc8d[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:57,955 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:57,955 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,957 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,957 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:57,957 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:57,957 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:57,957 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,957 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:57,958 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,958 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:57,958 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:57,958 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:57,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:57,974 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:57,974 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:57,974 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:57,974 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:57,975 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:57,981 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,981 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:57,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:57,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,981 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:57,982 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:57,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:57,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,983 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:57,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,986 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,986 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,986 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:57,986 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:57,986 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:57,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:57,986 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:57,986 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:57,986 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:57,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:57,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:57,986 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:57,986 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:57,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:57,987 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:57,987 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:57,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:57,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:58,087 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:58,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:58,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:58,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:58,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:58,090 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:58,090 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:58,090 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:58,090 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:58,090 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,090 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,092 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,092 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:58,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:58,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:58,092 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,092 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:58,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:58,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:58,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:58,093 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:58,093 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:58,093 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:58,094 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:58,094 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:58,094 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:58,094 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:58,094 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:58,094 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:58,094 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:58,094 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:58,094 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,094 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:58,094 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:58,104 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,104 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,104 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,104 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:58,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:58,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72f561[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:58,106 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:58,106 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,107 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,107 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:58,107 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:58,107 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,107 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:58,107 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:58,109 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,109 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:58,109 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:58,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,109 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:58,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:58,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:58,111 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:58,111 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:58,111 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:58,111 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:58,111 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:58,113 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,113 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,113 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:58,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,113 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,114 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,117 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,117 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,117 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,117 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,118 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:58,117 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:58,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:58,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:58,118 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:58,118 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:58,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:58,118 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,118 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,118 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:58,118 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:58,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:58,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:58,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838898134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:58,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838898136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:58,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838898138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:58,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838898139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,144 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=609 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/133713e312ce4f29a0826ee7fd5af799 2023-03-26 13:53:58,218 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:58,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:58,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:58,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:58,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:58,222 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:58,222 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:58,222 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:58,222 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:58,222 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,222 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,224 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,224 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:58,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:58,225 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,225 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:58,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:58,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,225 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:58,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:58,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:58,226 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:58,226 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:58,226 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:58,226 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:58,226 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:58,226 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:58,226 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:58,226 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:58,226 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:58,226 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:58,226 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:58,226 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,226 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:58,226 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:58,228 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,229 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,229 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:58,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c700564[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:58,232 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:58,232 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,233 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,233 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:58,233 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:58,233 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,233 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:58,233 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:58,235 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:58,235 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:58,235 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,235 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:58,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:58,238 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:58,238 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:58,238 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:58,238 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:58,238 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:58,239 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,239 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,239 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:58,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,241 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,243 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,245 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,245 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,245 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:58,246 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:58,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:58,246 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:58,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:58,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:58,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:58,245 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,246 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,246 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:58,246 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:58,246 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:58,246 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:58,246 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,346 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:58,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:58,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:58,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:58,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:58,353 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:58,354 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:58,354 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:58,354 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:58,354 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,354 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,355 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:58,355 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:58,356 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,356 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,357 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:58,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:58,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:58,357 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:58,357 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:58,357 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:58,357 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:58,357 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:58,357 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:58,357 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:58,357 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:58,357 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:58,358 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:58,358 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:58,358 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,358 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:58,358 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:58,360 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,360 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,361 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,361 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:58,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:58,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@235de9a2[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:58,362 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:58,362 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,364 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,364 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:58,364 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:58,364 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:58,364 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,364 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:58,365 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:58,365 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:58,365 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,365 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:58,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:58,368 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:58,368 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:58,368 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:58,368 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:58,368 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:58,370 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,370 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,370 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:58,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,371 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,376 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,383 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,384 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,384 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,384 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:58,384 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:58,384 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:58,384 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:58,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:58,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:58,384 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,385 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,385 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:58,385 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:58,385 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:58,385 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:58,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:58,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838898438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:58,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838898442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:58,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838898445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:58,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838898446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,486 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:58,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:58,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:58,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:58,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:58,491 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:58,491 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:58,492 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:58,492 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:58,492 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,492 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,493 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:58,493 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:58,494 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,494 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:58,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:58,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:58,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:58,496 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:58,496 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:58,496 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:58,502 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:58,502 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:58,502 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:58,502 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:58,502 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:58,504 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,504 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,504 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,504 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:58,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72dfaad0[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:58,509 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:58,509 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,510 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,510 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:58,510 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:58,510 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:58,510 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,511 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:58,516 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,516 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:58,516 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,516 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:58,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:58,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:58,518 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:58,518 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:58,518 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:58,519 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:58,519 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:58,520 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,520 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,520 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:58,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,521 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,522 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,525 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,525 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:58,525 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:58,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:58,525 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,526 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,526 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,526 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,526 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,525 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:58,526 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:58,526 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:58,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:58,525 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:58,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:58,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:58,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:58,560 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=609 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/a01647b3dcef46c281bc764c5277ef94 2023-03-26 13:53:58,626 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:58,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:58,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:58,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:58,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:58,630 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:58,630 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:58,630 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:58,630 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:58,632 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,632 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,636 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:58,636 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:58,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:58,637 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,637 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:58,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:58,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:58,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:58,638 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:58,638 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:58,638 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:58,638 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:58,638 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:58,638 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:58,638 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:58,638 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:58,638 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:58,638 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:58,638 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:58,638 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,639 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:58,639 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:58,641 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,641 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,642 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,642 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:58,643 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:58,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65118017[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:58,643 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,645 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,645 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:58,645 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,645 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:58,645 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:58,645 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:58,647 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,647 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:58,647 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,647 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:58,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:58,649 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:58,649 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:58,649 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:58,649 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:58,649 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:58,651 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,651 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,651 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:58,652 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,653 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,656 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:58,656 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:58,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:58,656 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:58,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:58,656 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,656 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,656 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,656 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,656 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,656 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,656 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:58,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:58,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:58,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:58,657 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:58,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:58,657 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:58,757 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:58,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:58,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:58,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:58,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:58,761 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:58,761 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:58,761 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:58,761 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:58,762 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,762 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,763 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,764 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:58,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:58,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:58,766 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,766 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:58,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:58,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:58,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:58,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:58,767 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:58,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:58,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:58,768 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:58,768 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:58,768 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:58,768 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:58,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:58,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:58,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:58,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:58,779 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,779 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,780 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,780 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:58,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:58,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9a6f2f4[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:58,781 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:58,781 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,782 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,782 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:58,782 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:58,782 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:58,782 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,782 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:58,784 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:58,784 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,784 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:58,784 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:58,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:58,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:58,786 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:58,787 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:58,787 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:58,787 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:58,787 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:58,789 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,789 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,789 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:58,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,790 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,791 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,793 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,793 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:58,793 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,793 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:58,793 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:58,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:58,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:58,793 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:58,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:58,793 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,793 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,793 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:58,794 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:58,793 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,794 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:58,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:58,894 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:58,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:58,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:58,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:58,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:58,897 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:58,897 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:58,897 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:58,897 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:58,898 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,898 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,899 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:58,899 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,899 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,899 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:58,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:58,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,900 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:58,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:58,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:58,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:58,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:58,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:58,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:58,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:58,900 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:58,900 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:58,901 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. as already flushing 2023-03-26 13:53:58,901 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:58,901 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:58,901 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:58,901 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,901 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:58,901 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:58,903 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,903 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,903 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,903 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:58,904 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:58,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@457a0a9b[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:58,904 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,907 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,907 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,907 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:58,907 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:58,907 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:58,907 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:58,909 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,909 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:58,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,909 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:58,909 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:58,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:58,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:58,911 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:58,911 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:58,911 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:58,911 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:58,911 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:58,919 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,919 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:58,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,919 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:58,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:58,920 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:58,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,925 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:58,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:58,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,934 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,934 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:58,934 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:58,934 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:58,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:58,934 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:58,934 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:58,934 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,934 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:58,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:58,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:53:58,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:58,935 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:58,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:58,935 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:58,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:58,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:58,947 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:58,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39074 deadline: 1679838898944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,952 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:58,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39082 deadline: 1679838898950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:58,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39064 deadline: 1679838898951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:53:58,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39056 deadline: 1679838898954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:58,972 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=609 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/13d34dd19ea54579b729ff74d2494e20 2023-03-26 13:53:58,978 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/133713e312ce4f29a0826ee7fd5af799 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/133713e312ce4f29a0826ee7fd5af799 2023-03-26 13:53:58,982 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/133713e312ce4f29a0826ee7fd5af799, entries=150, sequenceid=609, filesize=11.9 K 2023-03-26 13:53:58,983 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/a01647b3dcef46c281bc764c5277ef94 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/a01647b3dcef46c281bc764c5277ef94 2023-03-26 13:53:58,989 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/a01647b3dcef46c281bc764c5277ef94, entries=150, sequenceid=609, filesize=11.9 K 2023-03-26 13:53:58,990 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/13d34dd19ea54579b729ff74d2494e20 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/13d34dd19ea54579b729ff74d2494e20 2023-03-26 13:53:58,995 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/13d34dd19ea54579b729ff74d2494e20, entries=150, sequenceid=609, filesize=11.9 K 2023-03-26 13:53:58,995 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for c54bfce2a8106a499411906befe41bd8 in 1267ms, sequenceid=609, compaction requested=true 2023-03-26 13:53:58,995 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:58,996 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:58,996 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 13:53:58,996 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:58,996 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 13:53:58,996 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:58,998 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74342 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 13:53:58,998 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74376 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 13:53:58,998 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/A is initiating minor compaction (all files) 2023-03-26 13:53:58,998 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/B is initiating minor compaction (all files) 2023-03-26 13:53:58,998 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/A in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:58,998 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/B in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:58,998 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1ad273f134e349c49539a1496feb965a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1cabde85fd1547fb93f275c581dff7fa, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/2a317b7b71774ea091e236198ac9edf5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/b3d2c2e96d5d45e3a5c288529c7d913a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/d531fa34f632475996ab547785186a6e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/133713e312ce4f29a0826ee7fd5af799] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=72.6 K 2023-03-26 13:53:58,998 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/8c580fea9dea44fcaf8dd033d64a6b09, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/896f0656f1594333b101e9a24a3ae17a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/f2a7fc5080a34902a5ec19455c5862bc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e12c5fbf12b345e09ac552910b5b3745, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/22876eb85bfa4900823f06c41d6cf471, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/a01647b3dcef46c281bc764c5277ef94] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=72.6 K 2023-03-26 13:53:58,999 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 1ad273f134e349c49539a1496feb965a, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1679838833879 2023-03-26 13:53:58,999 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 8c580fea9dea44fcaf8dd033d64a6b09, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1679838833879 2023-03-26 13:53:58,999 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 1cabde85fd1547fb93f275c581dff7fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1679838834204 2023-03-26 13:53:58,999 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 896f0656f1594333b101e9a24a3ae17a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1679838834204 2023-03-26 13:53:59,000 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 2a317b7b71774ea091e236198ac9edf5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1679838834847 2023-03-26 13:53:59,000 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting b3d2c2e96d5d45e3a5c288529c7d913a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1679838835468 2023-03-26 13:53:59,000 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting f2a7fc5080a34902a5ec19455c5862bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1679838834847 2023-03-26 13:53:59,001 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e12c5fbf12b345e09ac552910b5b3745, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1679838835468 2023-03-26 13:53:59,001 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting d531fa34f632475996ab547785186a6e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=598, earliestPutTs=1679838836590 2023-03-26 13:53:59,002 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 22876eb85bfa4900823f06c41d6cf471, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=598, earliestPutTs=1679838836590 2023-03-26 13:53:59,002 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 133713e312ce4f29a0826ee7fd5af799, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=609, earliestPutTs=1679838837723 2023-03-26 13:53:59,002 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting a01647b3dcef46c281bc764c5277ef94, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=609, earliestPutTs=1679838837723 2023-03-26 13:53:59,026 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#B#compaction#126 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:59,033 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#A#compaction#127 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:59,035 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:53:59,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:53:59,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:53:59,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:53:59,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:53:59,039 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:53:59,039 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:59,039 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:53:59,039 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:53:59,040 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:59,040 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:59,042 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:59,042 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:59,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:59,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:59,042 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:59,042 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:53:59,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:59,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:53:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:59,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:53:59,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:53:59,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:53:59,048 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:53:59,048 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:53:59,048 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:53:59,048 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:53:59,048 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:53:59,049 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:59,049 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. started... 2023-03-26 13:53:59,049 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-26 13:53:59,050 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:53:59,050 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:59,050 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:53:59,050 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:59,050 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:53:59,050 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:53:59,061 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/5e877fa51b5c48babf9e2edf911ec0af 2023-03-26 13:53:59,456 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/df353dd0f356497bb29a485e6f1de846 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/df353dd0f356497bb29a485e6f1de846 2023-03-26 13:53:59,457 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/0208c6e0699d428eaf7a63d93cc3d9e4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/0208c6e0699d428eaf7a63d93cc3d9e4 2023-03-26 13:53:59,463 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c54bfce2a8106a499411906befe41bd8/A of c54bfce2a8106a499411906befe41bd8 into 0208c6e0699d428eaf7a63d93cc3d9e4(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:59,463 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:59,463 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/A, priority=10, startTime=1679838838996; duration=0sec 2023-03-26 13:53:59,464 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:53:59,464 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 13:53:59,466 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c54bfce2a8106a499411906befe41bd8/B of c54bfce2a8106a499411906befe41bd8 into df353dd0f356497bb29a485e6f1de846(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:59,466 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:59,466 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/B, priority=10, startTime=1679838838996; duration=0sec 2023-03-26 13:53:59,466 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74308 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 13:53:59,466 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): c54bfce2a8106a499411906befe41bd8/C is initiating minor compaction (all files) 2023-03-26 13:53:59,466 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:59,466 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c54bfce2a8106a499411906befe41bd8/C in TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:59,466 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/da98c6bf700641a28dda7c8fa6405326, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/8ee877fc8ff44cc8821deffed71a4790, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/022d0627cde54e8f9eb3d097d7b23c35, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/39ad0277b2c8424fb05f4f464a001a0e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/73a4a240178f4c21ac4038e9922ff4bc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/13d34dd19ea54579b729ff74d2494e20] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp, totalSize=72.6 K 2023-03-26 13:53:59,467 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting da98c6bf700641a28dda7c8fa6405326, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=528, earliestPutTs=1679838833879 2023-03-26 13:53:59,467 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 8ee877fc8ff44cc8821deffed71a4790, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=550, earliestPutTs=1679838834204 2023-03-26 13:53:59,468 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 022d0627cde54e8f9eb3d097d7b23c35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1679838834847 2023-03-26 13:53:59,468 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 39ad0277b2c8424fb05f4f464a001a0e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=584, earliestPutTs=1679838835468 2023-03-26 13:53:59,469 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 73a4a240178f4c21ac4038e9922ff4bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=598, earliestPutTs=1679838836590 2023-03-26 13:53:59,469 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 13d34dd19ea54579b729ff74d2494e20, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=609, earliestPutTs=1679838837723 2023-03-26 13:53:59,481 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/e2077c5c9a9e4c44b8a5e06944716776 2023-03-26 13:53:59,497 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): c54bfce2a8106a499411906befe41bd8#C#compaction#131 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:53:59,518 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/dff27b3e4473469ebc49abaf58ac009f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/dff27b3e4473469ebc49abaf58ac009f 2023-03-26 13:53:59,523 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in c54bfce2a8106a499411906befe41bd8/C of c54bfce2a8106a499411906befe41bd8 into dff27b3e4473469ebc49abaf58ac009f(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:53:59,523 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:59,524 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8., storeName=c54bfce2a8106a499411906befe41bd8/C, priority=10, startTime=1679838838996; duration=0sec 2023-03-26 13:53:59,524 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:53:59,542 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x64804459 to 127.0.0.1:54332 2023-03-26 13:53:59,542 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:53:59,544 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fcbe28d to 127.0.0.1:54332 2023-03-26 13:53:59,544 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3c9ded19 to 127.0.0.1:54332 2023-03-26 13:53:59,544 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:53:59,544 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:53:59,544 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x142b9819 to 127.0.0.1:54332 2023-03-26 13:53:59,545 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:53:59,907 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/90000e45a70d4bd9a5fb2619ffab471c 2023-03-26 13:53:59,912 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/5e877fa51b5c48babf9e2edf911ec0af as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/5e877fa51b5c48babf9e2edf911ec0af 2023-03-26 13:53:59,916 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/5e877fa51b5c48babf9e2edf911ec0af, entries=150, sequenceid=644, filesize=11.9 K 2023-03-26 13:53:59,916 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/e2077c5c9a9e4c44b8a5e06944716776 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e2077c5c9a9e4c44b8a5e06944716776 2023-03-26 13:53:59,920 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e2077c5c9a9e4c44b8a5e06944716776, entries=150, sequenceid=644, filesize=11.9 K 2023-03-26 13:53:59,921 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/90000e45a70d4bd9a5fb2619ffab471c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/90000e45a70d4bd9a5fb2619ffab471c 2023-03-26 13:53:59,925 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/90000e45a70d4bd9a5fb2619ffab471c, entries=150, sequenceid=644, filesize=11.9 K 2023-03-26 13:53:59,925 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=0 B/0 for c54bfce2a8106a499411906befe41bd8 in 876ms, sequenceid=644, compaction requested=false 2023-03-26 13:53:59,926 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:53:59,926 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:53:59,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:53:59,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:53:59,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:59,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:53:59,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:53:59,927 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:59,928 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:59,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:59,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:59,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:59,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:59,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:53:59,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:59,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:59,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:59,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:59,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:59,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:53:59,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a7ca1ce[Count = 0] remaining members to acquire global barrier 2023-03-26 13:53:59,929 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:53:59,929 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:59,930 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:59,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:59,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:59,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:53:59,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:53:59,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:53:59,931 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:59,931 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:53:59,932 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:53:59,932 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:59,932 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:59,932 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:53:59,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:59,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:59,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:59,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:59,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:59,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:59,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:59,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:59,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:59,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:59,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:53:59,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:53:59,934 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:53:59,934 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:53:59,934 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:53:59,934 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:53:59,934 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:53:59,937 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:59,937 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:59,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:59,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:59,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:53:59,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:53:59,937 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:59,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:59,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:53:59,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:59,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:59,937 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:53:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:59,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:59,938 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:53:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:53:59,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:59,941 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:59,941 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:53:59,941 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:59,941 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:53:59,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:53:59,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:53:59,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:53:59,941 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:53:59,941 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:53:59,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:53:59,941 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:53:59,941 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:53:59,941 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:59,941 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:53:59,941 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:53:59,941 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:53:59,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:53:59,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:53:59,956 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2e2005d0 to 127.0.0.1:54332 2023-03-26 13:53:59,956 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:53:59,959 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x79fee867 to 127.0.0.1:54332 2023-03-26 13:53:59,959 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:53:59,961 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e82daee to 127.0.0.1:54332 2023-03-26 13:53:59,961 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:53:59,966 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36dc7f7f to 127.0.0.1:54332 2023-03-26 13:53:59,966 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:00,041 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:00,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:06,643 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x75f5a5e6 to 127.0.0.1:54332 2023-03-26 13:54:06,643 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:06,643 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-26 13:54:06,643 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 106 2023-03-26 13:54:06,643 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 127 2023-03-26 13:54:06,643 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 117 2023-03-26 13:54:06,644 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 121 2023-03-26 13:54:06,644 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 47 2023-03-26 13:54:06,644 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-26 13:54:06,644 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(398): read 4300 2023-03-26 13:54:06,644 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(398): read 4346 2023-03-26 13:54:06,644 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-26 13:54:06,644 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(402): scanned 1910 2023-03-26 13:54:06,644 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(403): verified 5729 rows 2023-03-26 13:54:06,644 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(402): scanned 1920 2023-03-26 13:54:06,644 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(403): verified 5750 rows 2023-03-26 13:54:06,644 INFO [Listener at localhost/41531] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-26 13:54:06,644 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23162110 to 127.0.0.1:54332 2023-03-26 13:54:06,644 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:06,650 INFO [Listener at localhost/41531] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-26 13:54:06,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-26 13:54:06,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-26 13:54:06,663 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838846663"}]},"ts":"1679838846663"} 2023-03-26 13:54:06,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-26 13:54:06,665 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-26 13:54:06,750 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-26 13:54:06,752 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c54bfce2a8106a499411906befe41bd8, UNASSIGN}] 2023-03-26 13:54:06,754 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c54bfce2a8106a499411906befe41bd8, UNASSIGN 2023-03-26 13:54:06,754 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=c54bfce2a8106a499411906befe41bd8, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:06,755 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838846754"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838846754"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838846754"}]},"ts":"1679838846754"} 2023-03-26 13:54:06,756 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:54:06,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-26 13:54:06,914 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c54bfce2a8106a499411906befe41bd8 2023-03-26 13:54:06,914 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c54bfce2a8106a499411906befe41bd8, disabling compactions & flushes 2023-03-26 13:54:06,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:54:06,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:54:06,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. after waiting 0 ms 2023-03-26 13:54:06,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:54:06,915 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing c54bfce2a8106a499411906befe41bd8 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-26 13:54:06,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=A 2023-03-26 13:54:06,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:06,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=B 2023-03-26 13:54:06,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:06,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c54bfce2a8106a499411906befe41bd8, store=C 2023-03-26 13:54:06,915 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:06,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-26 13:54:07,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-26 13:54:07,323 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=655 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/28ac676173294f748641c55a3cb21c5d 2023-03-26 13:54:07,734 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=655 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/41ecc4079a0045fdb7ee675a918fe451 2023-03-26 13:54:07,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-26 13:54:08,145 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=655 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/f5c2433175d647b18373fe3c13adbae2 2023-03-26 13:54:08,149 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/A/28ac676173294f748641c55a3cb21c5d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/28ac676173294f748641c55a3cb21c5d 2023-03-26 13:54:08,153 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/28ac676173294f748641c55a3cb21c5d, entries=100, sequenceid=655, filesize=9.5 K 2023-03-26 13:54:08,154 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/B/41ecc4079a0045fdb7ee675a918fe451 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/41ecc4079a0045fdb7ee675a918fe451 2023-03-26 13:54:08,158 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/41ecc4079a0045fdb7ee675a918fe451, entries=100, sequenceid=655, filesize=9.5 K 2023-03-26 13:54:08,159 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/.tmp/C/f5c2433175d647b18373fe3c13adbae2 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/f5c2433175d647b18373fe3c13adbae2 2023-03-26 13:54:08,163 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/f5c2433175d647b18373fe3c13adbae2, entries=100, sequenceid=655, filesize=9.5 K 2023-03-26 13:54:08,164 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for c54bfce2a8106a499411906befe41bd8 in 1249ms, sequenceid=655, compaction requested=true 2023-03-26 13:54:08,185 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/cfe19306a224425eac61309608def4aa, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/33836bb6787c43c2969df6e890970e8b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/df64391b6e3c42aca27635cae2019273, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/99e1618bda1146a28a8656b8a9073b55, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/242e392f053041a9b9c3ea52eeccf17e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/5072ab0a565d49679f215d5f5d19ac22, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/31ed28ae2c7b4af29ff994fd049eb89b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1618bf3e7fc144bfa7e5c408f5165d6a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/652252ed131c43dcb06a3dd3e95a96d3, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/f3aec82c6d7045b7b519992aa92ec5fb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/f2104fee5c1b476cafec4bd059532cb2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/0ab8531594b24d5fa50534d016be5f4f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e556cfe48d0940ccaca5e6733ac0b563, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e21393b7af614635a6b925c1214c9a9c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e41d4c0a3a7b4cfea47b72c9d01a66a9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/470f2b5f49124218adb567b4141d3311, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e1cad71b100e4832b4a61e2e47fb1622, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/026b7695d64246139329ef45d1e960a9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/a59e2b751b60431684f290cd90d11412, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/87dbc2256ca04fcb83683dc95bd69c4e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/bf6497e677014c8b8c880b3c4e6d018e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/275d709fb2b446ec9d14eee31063ae44, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/433589000cc34e38831fa0c212a50de1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/02290a35dfac4f4b96d0ea42399e1fb2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/f4fa595b83284f52839e314af43d1886, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/82366c9d993b4675a23ba8a0485f1674, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/8ad2c36079404498a0395167d936a767, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/7d36509044cc42a783efb4a3649705d9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/74eb06fc70024b138ed307c0f0cb2bed, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/3e135763009a4fd39321f6318cd61e04, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/cf49ed7226a84ca495773eebae7b2c4e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1fa60de424bd434caabbed8657aec23d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/62f0c4eca80c41f7b4b9b5fdc945e6f6, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/29d98a4d16cc41b38d947afc1f1b3958, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/4c017c3cc0f24ea4af4b049966904ed7, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/8ec9a13eabf34a21a1747a7a78dc0a24, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1ad273f134e349c49539a1496feb965a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1cabde85fd1547fb93f275c581dff7fa, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/2a317b7b71774ea091e236198ac9edf5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/b3d2c2e96d5d45e3a5c288529c7d913a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/d531fa34f632475996ab547785186a6e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/133713e312ce4f29a0826ee7fd5af799] to archive 2023-03-26 13:54:08,187 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 13:54:08,193 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/cfe19306a224425eac61309608def4aa to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/cfe19306a224425eac61309608def4aa 2023-03-26 13:54:08,195 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/33836bb6787c43c2969df6e890970e8b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/33836bb6787c43c2969df6e890970e8b 2023-03-26 13:54:08,196 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/df64391b6e3c42aca27635cae2019273 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/df64391b6e3c42aca27635cae2019273 2023-03-26 13:54:08,197 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/99e1618bda1146a28a8656b8a9073b55 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/99e1618bda1146a28a8656b8a9073b55 2023-03-26 13:54:08,199 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/242e392f053041a9b9c3ea52eeccf17e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/242e392f053041a9b9c3ea52eeccf17e 2023-03-26 13:54:08,200 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/5072ab0a565d49679f215d5f5d19ac22 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/5072ab0a565d49679f215d5f5d19ac22 2023-03-26 13:54:08,202 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/31ed28ae2c7b4af29ff994fd049eb89b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/31ed28ae2c7b4af29ff994fd049eb89b 2023-03-26 13:54:08,203 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1618bf3e7fc144bfa7e5c408f5165d6a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1618bf3e7fc144bfa7e5c408f5165d6a 2023-03-26 13:54:08,204 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/652252ed131c43dcb06a3dd3e95a96d3 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/652252ed131c43dcb06a3dd3e95a96d3 2023-03-26 13:54:08,206 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/f3aec82c6d7045b7b519992aa92ec5fb to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/f3aec82c6d7045b7b519992aa92ec5fb 2023-03-26 13:54:08,207 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/f2104fee5c1b476cafec4bd059532cb2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/f2104fee5c1b476cafec4bd059532cb2 2023-03-26 13:54:08,209 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/0ab8531594b24d5fa50534d016be5f4f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/0ab8531594b24d5fa50534d016be5f4f 2023-03-26 13:54:08,210 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e556cfe48d0940ccaca5e6733ac0b563 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e556cfe48d0940ccaca5e6733ac0b563 2023-03-26 13:54:08,211 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e21393b7af614635a6b925c1214c9a9c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e21393b7af614635a6b925c1214c9a9c 2023-03-26 13:54:08,213 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e41d4c0a3a7b4cfea47b72c9d01a66a9 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e41d4c0a3a7b4cfea47b72c9d01a66a9 2023-03-26 13:54:08,214 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/470f2b5f49124218adb567b4141d3311 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/470f2b5f49124218adb567b4141d3311 2023-03-26 13:54:08,216 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e1cad71b100e4832b4a61e2e47fb1622 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/e1cad71b100e4832b4a61e2e47fb1622 2023-03-26 13:54:08,217 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/026b7695d64246139329ef45d1e960a9 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/026b7695d64246139329ef45d1e960a9 2023-03-26 13:54:08,218 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/a59e2b751b60431684f290cd90d11412 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/a59e2b751b60431684f290cd90d11412 2023-03-26 13:54:08,219 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/87dbc2256ca04fcb83683dc95bd69c4e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/87dbc2256ca04fcb83683dc95bd69c4e 2023-03-26 13:54:08,220 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/bf6497e677014c8b8c880b3c4e6d018e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/bf6497e677014c8b8c880b3c4e6d018e 2023-03-26 13:54:08,221 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/275d709fb2b446ec9d14eee31063ae44 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/275d709fb2b446ec9d14eee31063ae44 2023-03-26 13:54:08,223 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/433589000cc34e38831fa0c212a50de1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/433589000cc34e38831fa0c212a50de1 2023-03-26 13:54:08,224 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/02290a35dfac4f4b96d0ea42399e1fb2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/02290a35dfac4f4b96d0ea42399e1fb2 2023-03-26 13:54:08,225 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/f4fa595b83284f52839e314af43d1886 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/f4fa595b83284f52839e314af43d1886 2023-03-26 13:54:08,226 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/82366c9d993b4675a23ba8a0485f1674 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/82366c9d993b4675a23ba8a0485f1674 2023-03-26 13:54:08,227 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/8ad2c36079404498a0395167d936a767 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/8ad2c36079404498a0395167d936a767 2023-03-26 13:54:08,228 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/7d36509044cc42a783efb4a3649705d9 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/7d36509044cc42a783efb4a3649705d9 2023-03-26 13:54:08,229 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/74eb06fc70024b138ed307c0f0cb2bed to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/74eb06fc70024b138ed307c0f0cb2bed 2023-03-26 13:54:08,231 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/3e135763009a4fd39321f6318cd61e04 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/3e135763009a4fd39321f6318cd61e04 2023-03-26 13:54:08,232 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/cf49ed7226a84ca495773eebae7b2c4e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/cf49ed7226a84ca495773eebae7b2c4e 2023-03-26 13:54:08,233 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1fa60de424bd434caabbed8657aec23d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1fa60de424bd434caabbed8657aec23d 2023-03-26 13:54:08,234 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/62f0c4eca80c41f7b4b9b5fdc945e6f6 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/62f0c4eca80c41f7b4b9b5fdc945e6f6 2023-03-26 13:54:08,235 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/29d98a4d16cc41b38d947afc1f1b3958 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/29d98a4d16cc41b38d947afc1f1b3958 2023-03-26 13:54:08,237 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/4c017c3cc0f24ea4af4b049966904ed7 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/4c017c3cc0f24ea4af4b049966904ed7 2023-03-26 13:54:08,238 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/8ec9a13eabf34a21a1747a7a78dc0a24 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/8ec9a13eabf34a21a1747a7a78dc0a24 2023-03-26 13:54:08,239 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1ad273f134e349c49539a1496feb965a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1ad273f134e349c49539a1496feb965a 2023-03-26 13:54:08,240 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1cabde85fd1547fb93f275c581dff7fa to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/1cabde85fd1547fb93f275c581dff7fa 2023-03-26 13:54:08,241 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/2a317b7b71774ea091e236198ac9edf5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/2a317b7b71774ea091e236198ac9edf5 2023-03-26 13:54:08,242 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/b3d2c2e96d5d45e3a5c288529c7d913a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/b3d2c2e96d5d45e3a5c288529c7d913a 2023-03-26 13:54:08,244 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/d531fa34f632475996ab547785186a6e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/d531fa34f632475996ab547785186a6e 2023-03-26 13:54:08,245 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/133713e312ce4f29a0826ee7fd5af799 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/133713e312ce4f29a0826ee7fd5af799 2023-03-26 13:54:08,269 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e58a1937955b45b283728b077f1880ff, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/3896817d58af4e3893a8db480a9229f6, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/eef618848cec41c994118d92ae77ef18, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/06c6a4d76392467b9da4fdf6e2b90367, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/98e3dbc4641b436eb7049f32d6de1ec6, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/ba10a5fb8a554bbcb11742d75ee3e4ba, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/51a3643804da423b90a756bcac5a73a6, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/88344d7c986741149c0da35967a9faed, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/24b3f28bdd584253b10293d4bbad287e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/d500efd608ef44e0b61f0c0e8d8af035, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/c1c2fa799d294cb0ab56e0146d0c4b9f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/74042ed7fef547b1a408779e1e219845, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/282f8aaad2434ae78de29ad00f0a2865, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/26cb4f96448b40dd938697d478a3449c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/10f83c24bb174200bb34d1e447e7271b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/789e0c6d48a44ac1a96d4cd784646bc1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/7e63c6c93b9e4de6b23b032de3df64d5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/cd207e8f7d5744e7b394a3ac6bfef917, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/873b959ad55f40858dc949cb6d812bde, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/91d4e3c86c2042838c22d2a9dda90621, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/a05dc142bab644168c737351019ae34c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/d4be37d176fd48d8854ff1ac41ac43d6, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e15126c2469f4a72ac54484689209501, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/de1fd86df22d44a980017e3248339aac, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/95664e0ce60b4603990b777853936085, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/73672e29d916475982245258bc84e08e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/9e2ca7cf4e7a4347972eebcd65633320, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/c6471d2600ac428c888b0ad6059aa92a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/2207660a7b3547589fe4d50fb98653c4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/24f5338722a8472a9fe223e7fd6efbbd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/3cd5368930bf447b83b4e0856113ac15, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/95b17d4c479d479cac7ba3b3b6e0f824, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/fc63ed77dfd3435893df50d979cb24a0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/f6cf98454d98418d92b10b950fa31df1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/f5e24985863943baa7020623cdd85437, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e175e428702c4d58b75b4f204631ed19, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/8c580fea9dea44fcaf8dd033d64a6b09, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/4374697ddaa246a1a3aa049fc48b0838, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/896f0656f1594333b101e9a24a3ae17a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/f2a7fc5080a34902a5ec19455c5862bc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e12c5fbf12b345e09ac552910b5b3745, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/22876eb85bfa4900823f06c41d6cf471, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/a01647b3dcef46c281bc764c5277ef94] to archive 2023-03-26 13:54:08,270 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 13:54:08,271 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e58a1937955b45b283728b077f1880ff to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e58a1937955b45b283728b077f1880ff 2023-03-26 13:54:08,273 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/3896817d58af4e3893a8db480a9229f6 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/3896817d58af4e3893a8db480a9229f6 2023-03-26 13:54:08,274 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/eef618848cec41c994118d92ae77ef18 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/eef618848cec41c994118d92ae77ef18 2023-03-26 13:54:08,275 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/06c6a4d76392467b9da4fdf6e2b90367 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/06c6a4d76392467b9da4fdf6e2b90367 2023-03-26 13:54:08,276 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/98e3dbc4641b436eb7049f32d6de1ec6 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/98e3dbc4641b436eb7049f32d6de1ec6 2023-03-26 13:54:08,277 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/ba10a5fb8a554bbcb11742d75ee3e4ba to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/ba10a5fb8a554bbcb11742d75ee3e4ba 2023-03-26 13:54:08,278 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/51a3643804da423b90a756bcac5a73a6 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/51a3643804da423b90a756bcac5a73a6 2023-03-26 13:54:08,279 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/88344d7c986741149c0da35967a9faed to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/88344d7c986741149c0da35967a9faed 2023-03-26 13:54:08,280 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/24b3f28bdd584253b10293d4bbad287e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/24b3f28bdd584253b10293d4bbad287e 2023-03-26 13:54:08,281 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/d500efd608ef44e0b61f0c0e8d8af035 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/d500efd608ef44e0b61f0c0e8d8af035 2023-03-26 13:54:08,283 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/c1c2fa799d294cb0ab56e0146d0c4b9f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/c1c2fa799d294cb0ab56e0146d0c4b9f 2023-03-26 13:54:08,284 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/74042ed7fef547b1a408779e1e219845 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/74042ed7fef547b1a408779e1e219845 2023-03-26 13:54:08,285 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/282f8aaad2434ae78de29ad00f0a2865 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/282f8aaad2434ae78de29ad00f0a2865 2023-03-26 13:54:08,286 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/26cb4f96448b40dd938697d478a3449c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/26cb4f96448b40dd938697d478a3449c 2023-03-26 13:54:08,287 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/10f83c24bb174200bb34d1e447e7271b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/10f83c24bb174200bb34d1e447e7271b 2023-03-26 13:54:08,288 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/789e0c6d48a44ac1a96d4cd784646bc1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/789e0c6d48a44ac1a96d4cd784646bc1 2023-03-26 13:54:08,289 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/7e63c6c93b9e4de6b23b032de3df64d5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/7e63c6c93b9e4de6b23b032de3df64d5 2023-03-26 13:54:08,290 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/cd207e8f7d5744e7b394a3ac6bfef917 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/cd207e8f7d5744e7b394a3ac6bfef917 2023-03-26 13:54:08,291 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/873b959ad55f40858dc949cb6d812bde to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/873b959ad55f40858dc949cb6d812bde 2023-03-26 13:54:08,292 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/91d4e3c86c2042838c22d2a9dda90621 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/91d4e3c86c2042838c22d2a9dda90621 2023-03-26 13:54:08,293 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/a05dc142bab644168c737351019ae34c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/a05dc142bab644168c737351019ae34c 2023-03-26 13:54:08,294 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/d4be37d176fd48d8854ff1ac41ac43d6 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/d4be37d176fd48d8854ff1ac41ac43d6 2023-03-26 13:54:08,295 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e15126c2469f4a72ac54484689209501 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e15126c2469f4a72ac54484689209501 2023-03-26 13:54:08,296 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/de1fd86df22d44a980017e3248339aac to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/de1fd86df22d44a980017e3248339aac 2023-03-26 13:54:08,297 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/95664e0ce60b4603990b777853936085 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/95664e0ce60b4603990b777853936085 2023-03-26 13:54:08,298 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/73672e29d916475982245258bc84e08e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/73672e29d916475982245258bc84e08e 2023-03-26 13:54:08,299 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/9e2ca7cf4e7a4347972eebcd65633320 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/9e2ca7cf4e7a4347972eebcd65633320 2023-03-26 13:54:08,300 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/c6471d2600ac428c888b0ad6059aa92a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/c6471d2600ac428c888b0ad6059aa92a 2023-03-26 13:54:08,301 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/2207660a7b3547589fe4d50fb98653c4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/2207660a7b3547589fe4d50fb98653c4 2023-03-26 13:54:08,302 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/24f5338722a8472a9fe223e7fd6efbbd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/24f5338722a8472a9fe223e7fd6efbbd 2023-03-26 13:54:08,303 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/3cd5368930bf447b83b4e0856113ac15 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/3cd5368930bf447b83b4e0856113ac15 2023-03-26 13:54:08,304 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/95b17d4c479d479cac7ba3b3b6e0f824 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/95b17d4c479d479cac7ba3b3b6e0f824 2023-03-26 13:54:08,305 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/fc63ed77dfd3435893df50d979cb24a0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/fc63ed77dfd3435893df50d979cb24a0 2023-03-26 13:54:08,306 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/f6cf98454d98418d92b10b950fa31df1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/f6cf98454d98418d92b10b950fa31df1 2023-03-26 13:54:08,307 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/f5e24985863943baa7020623cdd85437 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/f5e24985863943baa7020623cdd85437 2023-03-26 13:54:08,308 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e175e428702c4d58b75b4f204631ed19 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e175e428702c4d58b75b4f204631ed19 2023-03-26 13:54:08,309 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/8c580fea9dea44fcaf8dd033d64a6b09 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/8c580fea9dea44fcaf8dd033d64a6b09 2023-03-26 13:54:08,310 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/4374697ddaa246a1a3aa049fc48b0838 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/4374697ddaa246a1a3aa049fc48b0838 2023-03-26 13:54:08,311 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/896f0656f1594333b101e9a24a3ae17a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/896f0656f1594333b101e9a24a3ae17a 2023-03-26 13:54:08,312 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/f2a7fc5080a34902a5ec19455c5862bc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/f2a7fc5080a34902a5ec19455c5862bc 2023-03-26 13:54:08,313 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e12c5fbf12b345e09ac552910b5b3745 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e12c5fbf12b345e09ac552910b5b3745 2023-03-26 13:54:08,314 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/22876eb85bfa4900823f06c41d6cf471 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/22876eb85bfa4900823f06c41d6cf471 2023-03-26 13:54:08,315 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/a01647b3dcef46c281bc764c5277ef94 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/a01647b3dcef46c281bc764c5277ef94 2023-03-26 13:54:08,318 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/2c403239205c479c9de92d8d19e3eafc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e21a1bb11ce84ca2831379da85896b3b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/d43ee3f0619a449aa6158b78965c6bbb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/9ab0325e4a5441859cc3722b76a68ac4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/bfdbe64ec0da4223aeec5fcd8cb8dbf2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/d11453e7f7c24115a38969285da124d3, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/0f34332073924a738a8480807e4628c7, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/6d990461574149cd86c8c1e6b88d8501, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/7c800c043eb34517a9eb2963bee4126c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/7b560923a364478ab71d625dc5756b43, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/5d94b4012e5e45929696615a7fb57591, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/a9dd4396d2c541eeab3d0cdbe3cf8a49, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/ffba6244a7694d88a6eeb82362405844, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/4f6765dd3c8641fd90a27d8e6526cd20, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/3a75616ca9344f82946d81d412c8a980, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/f00d83c381c84a59be8c978dcd86a0ab, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e638a5c110284999a0b0206358a2b383, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/cda455e37f044645a987a211e2ff3416, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/8e30cc5b54d0441a95b00eeb56924aa5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/7a59aa5fae2e464dac471c4044c3d957, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/b2dbdb9460fa43c7865c00cc5e7eb885, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/a64848af322e495faa0357caeed8e137, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/747e2e608f3a4c64912b9841bd2ef98c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/8020f124f1a24a64b28d9940e6ce87e0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/9801fac668ff4fa1a948161982bf7221, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/ac6a9313f0ba4ec8b399cadce02f489a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/b5281181440d43b5812b1d14b3893945, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/56de2004a1114c67879dc2174e3e9e59, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/f1cdbb3327724c7381eb39c4a4c3cd8d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/3bf19f93ae884aff985300578a1f21ec, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e13b16c12cda44f89f568cab5c333299, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/09be9cf40b0542718c950c25e3ba69cd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/237bfb7f6e41499982d7be263bb588f9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/ebf78b54ecfb4e1498bd3cf5b04b1134, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/da98c6bf700641a28dda7c8fa6405326, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e6d50a4d0d364e95a9c1dd6bb762bb61, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/8ee877fc8ff44cc8821deffed71a4790, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/022d0627cde54e8f9eb3d097d7b23c35, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/39ad0277b2c8424fb05f4f464a001a0e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/73a4a240178f4c21ac4038e9922ff4bc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/13d34dd19ea54579b729ff74d2494e20] to archive 2023-03-26 13:54:08,319 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 13:54:08,321 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/2c403239205c479c9de92d8d19e3eafc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/2c403239205c479c9de92d8d19e3eafc 2023-03-26 13:54:08,322 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e21a1bb11ce84ca2831379da85896b3b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e21a1bb11ce84ca2831379da85896b3b 2023-03-26 13:54:08,323 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/d43ee3f0619a449aa6158b78965c6bbb to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/d43ee3f0619a449aa6158b78965c6bbb 2023-03-26 13:54:08,324 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/9ab0325e4a5441859cc3722b76a68ac4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/9ab0325e4a5441859cc3722b76a68ac4 2023-03-26 13:54:08,326 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/bfdbe64ec0da4223aeec5fcd8cb8dbf2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/bfdbe64ec0da4223aeec5fcd8cb8dbf2 2023-03-26 13:54:08,327 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/d11453e7f7c24115a38969285da124d3 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/d11453e7f7c24115a38969285da124d3 2023-03-26 13:54:08,328 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/0f34332073924a738a8480807e4628c7 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/0f34332073924a738a8480807e4628c7 2023-03-26 13:54:08,329 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/6d990461574149cd86c8c1e6b88d8501 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/6d990461574149cd86c8c1e6b88d8501 2023-03-26 13:54:08,330 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/7c800c043eb34517a9eb2963bee4126c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/7c800c043eb34517a9eb2963bee4126c 2023-03-26 13:54:08,332 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/7b560923a364478ab71d625dc5756b43 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/7b560923a364478ab71d625dc5756b43 2023-03-26 13:54:08,333 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/5d94b4012e5e45929696615a7fb57591 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/5d94b4012e5e45929696615a7fb57591 2023-03-26 13:54:08,335 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/a9dd4396d2c541eeab3d0cdbe3cf8a49 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/a9dd4396d2c541eeab3d0cdbe3cf8a49 2023-03-26 13:54:08,336 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/ffba6244a7694d88a6eeb82362405844 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/ffba6244a7694d88a6eeb82362405844 2023-03-26 13:54:08,337 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/4f6765dd3c8641fd90a27d8e6526cd20 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/4f6765dd3c8641fd90a27d8e6526cd20 2023-03-26 13:54:08,338 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/3a75616ca9344f82946d81d412c8a980 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/3a75616ca9344f82946d81d412c8a980 2023-03-26 13:54:08,339 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/f00d83c381c84a59be8c978dcd86a0ab to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/f00d83c381c84a59be8c978dcd86a0ab 2023-03-26 13:54:08,340 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e638a5c110284999a0b0206358a2b383 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e638a5c110284999a0b0206358a2b383 2023-03-26 13:54:08,341 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/cda455e37f044645a987a211e2ff3416 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/cda455e37f044645a987a211e2ff3416 2023-03-26 13:54:08,342 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/8e30cc5b54d0441a95b00eeb56924aa5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/8e30cc5b54d0441a95b00eeb56924aa5 2023-03-26 13:54:08,343 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/7a59aa5fae2e464dac471c4044c3d957 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/7a59aa5fae2e464dac471c4044c3d957 2023-03-26 13:54:08,344 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/b2dbdb9460fa43c7865c00cc5e7eb885 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/b2dbdb9460fa43c7865c00cc5e7eb885 2023-03-26 13:54:08,345 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/a64848af322e495faa0357caeed8e137 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/a64848af322e495faa0357caeed8e137 2023-03-26 13:54:08,346 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/747e2e608f3a4c64912b9841bd2ef98c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/747e2e608f3a4c64912b9841bd2ef98c 2023-03-26 13:54:08,347 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/8020f124f1a24a64b28d9940e6ce87e0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/8020f124f1a24a64b28d9940e6ce87e0 2023-03-26 13:54:08,348 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/9801fac668ff4fa1a948161982bf7221 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/9801fac668ff4fa1a948161982bf7221 2023-03-26 13:54:08,349 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/ac6a9313f0ba4ec8b399cadce02f489a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/ac6a9313f0ba4ec8b399cadce02f489a 2023-03-26 13:54:08,350 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/b5281181440d43b5812b1d14b3893945 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/b5281181440d43b5812b1d14b3893945 2023-03-26 13:54:08,351 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/56de2004a1114c67879dc2174e3e9e59 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/56de2004a1114c67879dc2174e3e9e59 2023-03-26 13:54:08,352 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/f1cdbb3327724c7381eb39c4a4c3cd8d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/f1cdbb3327724c7381eb39c4a4c3cd8d 2023-03-26 13:54:08,353 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/3bf19f93ae884aff985300578a1f21ec to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/3bf19f93ae884aff985300578a1f21ec 2023-03-26 13:54:08,355 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e13b16c12cda44f89f568cab5c333299 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e13b16c12cda44f89f568cab5c333299 2023-03-26 13:54:08,355 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/09be9cf40b0542718c950c25e3ba69cd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/09be9cf40b0542718c950c25e3ba69cd 2023-03-26 13:54:08,357 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/237bfb7f6e41499982d7be263bb588f9 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/237bfb7f6e41499982d7be263bb588f9 2023-03-26 13:54:08,358 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/ebf78b54ecfb4e1498bd3cf5b04b1134 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/ebf78b54ecfb4e1498bd3cf5b04b1134 2023-03-26 13:54:08,359 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/da98c6bf700641a28dda7c8fa6405326 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/da98c6bf700641a28dda7c8fa6405326 2023-03-26 13:54:08,360 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e6d50a4d0d364e95a9c1dd6bb762bb61 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/e6d50a4d0d364e95a9c1dd6bb762bb61 2023-03-26 13:54:08,361 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/8ee877fc8ff44cc8821deffed71a4790 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/8ee877fc8ff44cc8821deffed71a4790 2023-03-26 13:54:08,362 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/022d0627cde54e8f9eb3d097d7b23c35 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/022d0627cde54e8f9eb3d097d7b23c35 2023-03-26 13:54:08,363 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/39ad0277b2c8424fb05f4f464a001a0e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/39ad0277b2c8424fb05f4f464a001a0e 2023-03-26 13:54:08,364 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/73a4a240178f4c21ac4038e9922ff4bc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/73a4a240178f4c21ac4038e9922ff4bc 2023-03-26 13:54:08,365 DEBUG [StoreCloser-TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/13d34dd19ea54579b729ff74d2494e20 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/13d34dd19ea54579b729ff74d2494e20 2023-03-26 13:54:08,370 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/recovered.edits/658.seqid, newMaxSeqId=658, maxSeqId=1 2023-03-26 13:54:08,371 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8. 2023-03-26 13:54:08,371 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c54bfce2a8106a499411906befe41bd8: 2023-03-26 13:54:08,373 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c54bfce2a8106a499411906befe41bd8 2023-03-26 13:54:08,373 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=c54bfce2a8106a499411906befe41bd8, regionState=CLOSED 2023-03-26 13:54:08,373 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838848373"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679838848373"}]},"ts":"1679838848373"} 2023-03-26 13:54:08,376 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-03-26 13:54:08,376 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure c54bfce2a8106a499411906befe41bd8, server=jenkins-hbase4.apache.org,42517,1679838814721 in 1.6180 sec 2023-03-26 13:54:08,377 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-03-26 13:54:08,377 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c54bfce2a8106a499411906befe41bd8, UNASSIGN in 1.6240 sec 2023-03-26 13:54:08,378 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838848378"}]},"ts":"1679838848378"} 2023-03-26 13:54:08,379 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-26 13:54:08,380 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-26 13:54:08,382 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.7250 sec 2023-03-26 13:54:08,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-26 13:54:08,769 INFO [Listener at localhost/41531] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-03-26 13:54:08,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-26 13:54:08,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:54:08,781 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:54:08,782 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:54:08,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-26 13:54:08,787 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8 2023-03-26 13:54:08,791 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A, FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B, FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C, FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/recovered.edits] 2023-03-26 13:54:08,794 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/0208c6e0699d428eaf7a63d93cc3d9e4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/0208c6e0699d428eaf7a63d93cc3d9e4 2023-03-26 13:54:08,796 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/28ac676173294f748641c55a3cb21c5d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/28ac676173294f748641c55a3cb21c5d 2023-03-26 13:54:08,797 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/5e877fa51b5c48babf9e2edf911ec0af to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/A/5e877fa51b5c48babf9e2edf911ec0af 2023-03-26 13:54:08,800 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/41ecc4079a0045fdb7ee675a918fe451 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/41ecc4079a0045fdb7ee675a918fe451 2023-03-26 13:54:08,801 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/df353dd0f356497bb29a485e6f1de846 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/df353dd0f356497bb29a485e6f1de846 2023-03-26 13:54:08,803 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e2077c5c9a9e4c44b8a5e06944716776 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/B/e2077c5c9a9e4c44b8a5e06944716776 2023-03-26 13:54:08,806 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/90000e45a70d4bd9a5fb2619ffab471c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/90000e45a70d4bd9a5fb2619ffab471c 2023-03-26 13:54:08,807 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/dff27b3e4473469ebc49abaf58ac009f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/dff27b3e4473469ebc49abaf58ac009f 2023-03-26 13:54:08,809 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/f5c2433175d647b18373fe3c13adbae2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/C/f5c2433175d647b18373fe3c13adbae2 2023-03-26 13:54:08,812 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/recovered.edits/658.seqid to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8/recovered.edits/658.seqid 2023-03-26 13:54:08,812 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/c54bfce2a8106a499411906befe41bd8 2023-03-26 13:54:08,812 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 13:54:08,815 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:54:08,835 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-26 13:54:08,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-26 13:54:08,885 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-26 13:54:08,886 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:54:08,886 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-26 13:54:08,887 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679838848886"}]},"ts":"9223372036854775807"} 2023-03-26 13:54:08,889 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-26 13:54:08,889 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c54bfce2a8106a499411906befe41bd8, NAME => 'TestAcidGuarantees,,1679838818262.c54bfce2a8106a499411906befe41bd8.', STARTKEY => '', ENDKEY => ''}] 2023-03-26 13:54:08,889 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-26 13:54:08,889 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679838848889"}]},"ts":"9223372036854775807"} 2023-03-26 13:54:08,891 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-26 13:54:08,893 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:54:08,894 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 119 msec 2023-03-26 13:54:09,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-26 13:54:09,086 INFO [Listener at localhost/41531] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-03-26 13:54:09,103 INFO [Listener at localhost/41531] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=231 (was 208) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_284888099_15 at /127.0.0.1:35156 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0xcc5547b-shared-pool-11 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0xcc5547b-shared-pool-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:42517-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0xcc5547b-shared-pool-12 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0xcc5547b-shared-pool-10 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=545 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=434 (was 340) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 169) - ProcessCount LEAK? -, AvailableMemoryMB=9097 (was 9709) 2023-03-26 13:54:09,115 INFO [Listener at localhost/41531] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=231, OpenFileDescriptor=545, MaxFileDescriptor=60000, SystemLoadAverage=434, ProcessCount=172, AvailableMemoryMB=9096 2023-03-26 13:54:09,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-26 13:54:09,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 13:54:09,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-26 13:54:09,120 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-26 13:54:09,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-03-26 13:54:09,121 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-26 13:54:09,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-26 13:54:09,122 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:09,123 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80 empty. 2023-03-26 13:54:09,123 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:09,123 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 13:54:09,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-26 13:54:09,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-26 13:54:09,532 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-26 13:54:09,533 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a739eafaf8b68b3135ee013236515a80, NAME => 'TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp 2023-03-26 13:54:09,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-26 13:54:09,939 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:54:09,939 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a739eafaf8b68b3135ee013236515a80, disabling compactions & flushes 2023-03-26 13:54:09,939 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:09,939 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:09,939 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. after waiting 0 ms 2023-03-26 13:54:09,939 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:09,939 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:09,940 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:09,942 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-26 13:54:09,943 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838849943"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679838849943"}]},"ts":"1679838849943"} 2023-03-26 13:54:09,945 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-26 13:54:09,946 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-26 13:54:09,946 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838849946"}]},"ts":"1679838849946"} 2023-03-26 13:54:09,947 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-26 13:54:09,950 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a739eafaf8b68b3135ee013236515a80, ASSIGN}] 2023-03-26 13:54:09,952 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a739eafaf8b68b3135ee013236515a80, ASSIGN 2023-03-26 13:54:09,953 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a739eafaf8b68b3135ee013236515a80, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42517,1679838814721; forceNewPlan=false, retain=false 2023-03-26 13:54:10,104 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=a739eafaf8b68b3135ee013236515a80, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:10,104 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838850104"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838850104"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838850104"}]},"ts":"1679838850104"} 2023-03-26 13:54:10,105 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:54:10,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-26 13:54:10,261 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:10,261 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a739eafaf8b68b3135ee013236515a80, NAME => 'TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.', STARTKEY => '', ENDKEY => ''} 2023-03-26 13:54:10,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:10,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:54:10,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:10,262 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:10,263 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:10,265 DEBUG [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A 2023-03-26 13:54:10,265 DEBUG [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A 2023-03-26 13:54:10,265 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:54:10,265 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a739eafaf8b68b3135ee013236515a80 columnFamilyName A 2023-03-26 13:54:10,266 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] regionserver.HStore(310): Store=a739eafaf8b68b3135ee013236515a80/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:54:10,266 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:10,267 DEBUG [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B 2023-03-26 13:54:10,267 DEBUG [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B 2023-03-26 13:54:10,267 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:54:10,267 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a739eafaf8b68b3135ee013236515a80 columnFamilyName B 2023-03-26 13:54:10,268 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] regionserver.HStore(310): Store=a739eafaf8b68b3135ee013236515a80/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:54:10,268 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:10,269 DEBUG [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C 2023-03-26 13:54:10,269 DEBUG [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C 2023-03-26 13:54:10,269 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:54:10,269 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a739eafaf8b68b3135ee013236515a80 columnFamilyName C 2023-03-26 13:54:10,270 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] regionserver.HStore(310): Store=a739eafaf8b68b3135ee013236515a80/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:54:10,270 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:10,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:10,271 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:10,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 13:54:10,274 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:10,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-26 13:54:10,277 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a739eafaf8b68b3135ee013236515a80; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60107167, jitterRate=-0.10433341562747955}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 13:54:10,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:10,277 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., pid=18, masterSystemTime=1679838850257 2023-03-26 13:54:10,279 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:10,279 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:10,279 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=a739eafaf8b68b3135ee013236515a80, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:10,279 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838850279"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679838850279"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679838850279"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679838850279"}]},"ts":"1679838850279"} 2023-03-26 13:54:10,282 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-03-26 13:54:10,282 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 in 175 msec 2023-03-26 13:54:10,283 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-03-26 13:54:10,283 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a739eafaf8b68b3135ee013236515a80, ASSIGN in 332 msec 2023-03-26 13:54:10,283 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-26 13:54:10,284 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838850284"}]},"ts":"1679838850284"} 2023-03-26 13:54:10,285 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-26 13:54:10,287 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-26 13:54:10,288 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1700 sec 2023-03-26 13:54:11,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-26 13:54:11,227 INFO [Listener at localhost/41531] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-03-26 13:54:11,227 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x3ab850e7 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:11,234 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7053d0b0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:11,235 DEBUG [Listener at localhost/41531] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:11,239 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39984, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:11,240 DEBUG [Listener at localhost/41531] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-26 13:54:11,246 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:38586, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-26 13:54:11,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-26 13:54:11,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 13:54:11,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-26 13:54:11,673 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-26 13:54:11,674 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-26 13:54:11,674 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-26 13:54:11,678 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-26 13:54:11,690 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a739eafaf8b68b3135ee013236515a80, REOPEN/MOVE}] 2023-03-26 13:54:11,691 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a739eafaf8b68b3135ee013236515a80, REOPEN/MOVE 2023-03-26 13:54:11,692 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=a739eafaf8b68b3135ee013236515a80, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:11,692 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838851692"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838851692"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838851692"}]},"ts":"1679838851692"} 2023-03-26 13:54:11,693 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:54:11,845 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:11,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a739eafaf8b68b3135ee013236515a80, disabling compactions & flushes 2023-03-26 13:54:11,846 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:11,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:11,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. after waiting 0 ms 2023-03-26 13:54:11,846 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:11,850 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-26 13:54:11,851 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:11,851 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:11,851 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: a739eafaf8b68b3135ee013236515a80 to self. 2023-03-26 13:54:11,852 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:11,853 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=a739eafaf8b68b3135ee013236515a80, regionState=CLOSED 2023-03-26 13:54:11,853 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838851853"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679838851853"}]},"ts":"1679838851853"} 2023-03-26 13:54:11,856 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-03-26 13:54:11,856 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 in 161 msec 2023-03-26 13:54:11,856 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a739eafaf8b68b3135ee013236515a80, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,42517,1679838814721; forceNewPlan=false, retain=true 2023-03-26 13:54:12,007 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=a739eafaf8b68b3135ee013236515a80, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,007 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838852007"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838852007"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838852007"}]},"ts":"1679838852007"} 2023-03-26 13:54:12,009 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:54:12,164 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:12,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a739eafaf8b68b3135ee013236515a80, NAME => 'TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.', STARTKEY => '', ENDKEY => ''} 2023-03-26 13:54:12,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:12,164 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:54:12,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:12,165 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:12,167 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:12,169 DEBUG [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A 2023-03-26 13:54:12,169 DEBUG [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A 2023-03-26 13:54:12,169 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:54:12,172 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a739eafaf8b68b3135ee013236515a80 columnFamilyName A 2023-03-26 13:54:12,174 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] regionserver.HStore(310): Store=a739eafaf8b68b3135ee013236515a80/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:54:12,175 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:12,176 DEBUG [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B 2023-03-26 13:54:12,176 DEBUG [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B 2023-03-26 13:54:12,176 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:54:12,176 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a739eafaf8b68b3135ee013236515a80 columnFamilyName B 2023-03-26 13:54:12,177 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] regionserver.HStore(310): Store=a739eafaf8b68b3135ee013236515a80/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:54:12,177 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:12,178 DEBUG [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C 2023-03-26 13:54:12,178 DEBUG [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C 2023-03-26 13:54:12,178 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:54:12,178 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a739eafaf8b68b3135ee013236515a80 columnFamilyName C 2023-03-26 13:54:12,179 INFO [StoreOpener-a739eafaf8b68b3135ee013236515a80-1] regionserver.HStore(310): Store=a739eafaf8b68b3135ee013236515a80/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:54:12,179 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:12,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:12,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:12,182 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 13:54:12,183 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:12,184 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a739eafaf8b68b3135ee013236515a80; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63178429, jitterRate=-0.05856804549694061}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 13:54:12,184 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:12,185 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., pid=23, masterSystemTime=1679838852160 2023-03-26 13:54:12,186 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:12,186 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:12,187 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=a739eafaf8b68b3135ee013236515a80, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,187 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838852187"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679838852187"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679838852187"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679838852187"}]},"ts":"1679838852187"} 2023-03-26 13:54:12,189 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-03-26 13:54:12,189 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 in 179 msec 2023-03-26 13:54:12,191 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-03-26 13:54:12,191 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a739eafaf8b68b3135ee013236515a80, REOPEN/MOVE in 499 msec 2023-03-26 13:54:12,193 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-03-26 13:54:12,193 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 515 msec 2023-03-26 13:54:12,195 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 939 msec 2023-03-26 13:54:12,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-03-26 13:54:12,204 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x63ebf1e9 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:12,209 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@695af552, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:12,210 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x52d98fed to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:12,215 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52f29016, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:12,215 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x18496e17 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:12,219 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f13c14, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:12,219 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x25441895 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:12,224 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a5a81b5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:12,224 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x40e0a3dc to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:12,227 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45e59be5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:12,227 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x2668d41d to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:12,232 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fbb4986, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:12,233 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x6b416c44 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:12,236 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b69850d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:12,236 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x204b5880 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:12,240 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40883381, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:12,240 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x41173aec to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:12,244 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3be2e516, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:12,251 DEBUG [hconnection-0x665941a7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:12,251 DEBUG [hconnection-0x57102b15-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:12,251 DEBUG [hconnection-0x19fdb87d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:12,251 DEBUG [hconnection-0x7d6cbb11-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:12,252 DEBUG [hconnection-0x173fb09b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:12,256 DEBUG [hconnection-0x3bf29349-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:12,256 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39988, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:12,257 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40014, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:12,256 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40010, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:12,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:12,257 DEBUG [hconnection-0x44daf48-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:12,261 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40036, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:12,261 DEBUG [hconnection-0x23daec1b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:12,260 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39990, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:12,259 DEBUG [hconnection-0x173ee5da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:12,264 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40020, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:12,267 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40040, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:12,268 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40056, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:12,268 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:40054, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:12,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:12,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:12,284 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:12,284 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:12,284 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:12,284 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:12,290 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,290 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:12,293 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 13:54:12,293 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,293 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,293 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:12,294 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:12,295 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:12,295 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:12,295 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:12,295 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:12,295 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:12,304 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:12,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:12,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:12,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:12,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,305 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:12,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:12,308 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:12,308 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:12,308 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:12,312 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:12,312 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:12,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838912321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679838852307, max timestamp=1679838852328Num uniques 859; 2023-03-26 13:54:12,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 13:54:12,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679838852307, max timestamp=1679838852328Num uniques 859; 2023-03-26 13:54:12,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 13:54:12,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679838852307, max timestamp=1679838852328Num uniques 859; 2023-03-26 13:54:12,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 13:54:12,336 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:12,337 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:12,337 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:12,337 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:12,337 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:12,337 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:12,337 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838912351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838912352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838912353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838912355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,375 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:12,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:12,377 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:12,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:12,378 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,379 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:12,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:12,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@238491ae[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:12,380 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:12,380 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,381 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,381 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:12,381 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:12,381 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:12,383 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,384 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:12,384 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:12,384 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:12,384 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:12,384 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:12,384 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:12,384 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:12,384 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:12,384 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:12,384 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:12,394 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,394 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,394 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:12,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:12,395 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:12,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,396 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,399 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,399 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,399 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:12,399 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:12,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:12,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:12,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:12,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:12,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:12,399 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:12,399 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,400 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,400 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,400 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,400 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,400 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:12,400 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:12,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:12,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:12,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838912436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838912457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838912457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838912457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,461 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838912458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,500 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:12,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:12,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:12,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:12,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:12,538 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:12,538 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:12,538 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:12,538 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:12,538 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,539 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,541 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,541 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:12,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:12,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:12,541 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,541 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:12,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:12,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:12,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:12,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:12,552 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:12,552 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:12,552 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:12,560 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:12,560 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:12,563 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:12,563 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:12,563 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:12,563 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:12,563 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:12,563 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:12,563 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,563 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:12,563 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:12,566 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,566 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:12,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:12,566 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,567 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:12,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:12,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:12,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d0bf678[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:12,569 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:12,570 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,571 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:12,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:12,571 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,571 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:12,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:12,574 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:12,574 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:12,574 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:12,574 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:12,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:12,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:12,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:12,578 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:12,578 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:12,578 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:12,578 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:12,578 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:12,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:12,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:12,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:12,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:12,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:12,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:12,581 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:12,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,582 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:12,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,587 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:12,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:12,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:12,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:12,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:12,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:12,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:12,587 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:12,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,587 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:12,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:12,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:12,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838912639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838912662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838912663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838912664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838912664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,687 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:12,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:12,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:12,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:12,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:12,700 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:12,700 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:12,700 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:12,700 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:12,701 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,701 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,702 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:12,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:12,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:12,702 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,703 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,703 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:12,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:12,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:12,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:12,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:12,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:12,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:12,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:12,712 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:12,712 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:12,712 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:12,712 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:12,712 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:12,712 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:12,712 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:12,712 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:12,712 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,720 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:12,720 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:12,723 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:12,723 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,724 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,724 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:12,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:12,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:12,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:12,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8b01990[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:12,726 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:12,726 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,727 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,727 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:12,727 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:12,727 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:12,728 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,728 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:12,730 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:12,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:12,733 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:12,733 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:12,733 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:12,734 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:12,734 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:12,735 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,735 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,735 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:12,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:12,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,738 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,740 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,742 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:12,742 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:12,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:12,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:12,742 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:12,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:12,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:12,742 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,743 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,743 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,743 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,743 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:12,743 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:12,743 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,743 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,743 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:12,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:12,791 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326ab2c5e4889a94117a7d3a4f6dd05cf5f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ab2c5e4889a94117a7d3a4f6dd05cf5f 2023-03-26 13:54:12,843 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:12,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:12,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:12,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:12,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:12,847 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:12,847 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:12,847 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:12,847 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:12,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,850 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:12,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:12,850 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:12,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,850 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,850 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:12,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:12,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:12,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:12,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:12,856 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:12,856 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:12,856 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:12,858 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:12,858 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:12,858 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:12,858 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:12,858 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:12,858 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:12,860 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:12,860 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:12,860 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,860 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:12,860 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:12,867 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,867 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:12,868 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,868 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:12,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:12,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:12,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:12,870 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:12,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27edbb99[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:12,870 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,871 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,871 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:12,871 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:12,871 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:12,871 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,872 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:12,874 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:12,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:12,874 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:12,874 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:12,874 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:12,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:12,878 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:12,878 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:12,878 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:12,878 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:12,878 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:12,880 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,880 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:12,880 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:12,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:12,880 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:12,882 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,887 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,887 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,887 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:12,887 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,888 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,888 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:12,887 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:12,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:12,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:12,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:12,888 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:12,888 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:12,888 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:12,888 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:12,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:12,888 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:12,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:12,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:12,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838912943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838912964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838912966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838912967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:12,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838912971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:12,988 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:12,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:12,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:13,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:13,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:13,004 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:13,004 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:13,004 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:13,004 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:13,005 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,005 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:13,006 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,006 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:13,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:13,007 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,007 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:13,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:13,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:13,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:13,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:13,012 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:13,012 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:13,012 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:13,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:13,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:13,021 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:13,021 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:13,021 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:13,021 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:13,021 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:13,021 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:13,021 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,021 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:13,021 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:13,023 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,023 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:13,024 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,024 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:13,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:13,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:13,025 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:13,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20e3572b[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:13,025 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,028 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,028 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:13,028 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:13,028 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:13,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:13,030 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:13,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:13,030 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:13,031 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:13,031 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:13,031 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:13,031 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:13,031 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:13,031 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:13,032 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,032 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:13,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,032 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:13,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:13,033 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:13,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:13,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:13,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:13,035 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:13,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:13,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,038 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,038 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:13,038 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:13,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:13,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:13,038 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:13,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:13,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:13,038 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:13,038 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:13,039 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,039 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:13,039 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:13,039 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,039 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,039 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:13,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:13,139 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:13,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:13,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:13,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:13,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:13,153 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:13,153 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:13,153 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:13,154 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:13,154 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,154 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:13,156 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:13,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:13,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:13,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,158 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,158 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:13,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:13,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:13,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:13,168 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:13,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:13,168 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:13,168 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:13,168 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:13,169 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:13,169 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:13,169 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:13,169 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:13,169 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:13,169 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:13,169 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:13,169 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:13,169 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,169 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:13,169 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:13,171 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,171 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:13,171 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,172 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:13,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:13,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6299dd79[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:13,173 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:13,173 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,175 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:13,175 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,175 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:13,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:13,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:13,177 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:13,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:13,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:13,177 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:13,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:13,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:13,180 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:13,181 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:13,181 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:13,181 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:13,181 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:13,184 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,184 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,184 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:13,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:13,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:13,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:13,185 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:13,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:13,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:13,186 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:13,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:13,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:13,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,188 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:13,189 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:13,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:13,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:13,189 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,189 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:13,189 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:13,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:13,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:13,189 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:13,189 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,189 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,189 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,189 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,189 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:13,189 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:13,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:13,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:13,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:13,203 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/63dd65ae30f64f0d8ea160ea9b8f55b9 2023-03-26 13:54:13,226 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/4fb2e4ac1db9405c8f93f262744b933c 2023-03-26 13:54:13,246 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/22e30b94be774f1092dbf9f2255e4cbb 2023-03-26 13:54:13,251 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/63dd65ae30f64f0d8ea160ea9b8f55b9 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/63dd65ae30f64f0d8ea160ea9b8f55b9 2023-03-26 13:54:13,256 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/63dd65ae30f64f0d8ea160ea9b8f55b9, entries=150, sequenceid=16, filesize=25.1 K 2023-03-26 13:54:13,257 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/4fb2e4ac1db9405c8f93f262744b933c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/4fb2e4ac1db9405c8f93f262744b933c 2023-03-26 13:54:13,268 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/4fb2e4ac1db9405c8f93f262744b933c, entries=150, sequenceid=16, filesize=11.6 K 2023-03-26 13:54:13,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/22e30b94be774f1092dbf9f2255e4cbb as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/22e30b94be774f1092dbf9f2255e4cbb 2023-03-26 13:54:13,277 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/22e30b94be774f1092dbf9f2255e4cbb, entries=150, sequenceid=16, filesize=11.6 K 2023-03-26 13:54:13,278 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for a739eafaf8b68b3135ee013236515a80 in 985ms, sequenceid=16, compaction requested=false 2023-03-26 13:54:13,278 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-26 13:54:13,279 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:13,289 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:13,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:13,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:13,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:13,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:13,296 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:13,296 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:13,296 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:13,296 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:13,297 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,297 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:13,299 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:13,299 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:13,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:13,300 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,300 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:13,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:13,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:13,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:13,301 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:13,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:13,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:13,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:13,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:13,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:13,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:13,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:13,304 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:13,304 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:13,304 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=127.47 KB heapSize=351 KB 2023-03-26 13:54:13,305 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:13,305 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:13,305 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:13,305 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:13,305 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:13,305 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:13,368 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326a1bd9b3c52aa430ba06c6bcdd9b0fae4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a1bd9b3c52aa430ba06c6bcdd9b0fae4 2023-03-26 13:54:13,415 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/34016a20526c400d9cdff5c0ced61960 2023-03-26 13:54:13,452 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:13,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:13,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:13,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838913475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:13,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838913476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:13,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838913477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:13,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838913477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:13,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838913481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:13,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838913580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:13,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838913582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:13,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838913582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:13,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838913587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:13,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838913782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:13,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838913783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,785 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:13,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838913784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:13,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838913790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,861 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/13dd805b5c7347e0ab11e3ab5da9fc9c 2023-03-26 13:54:13,919 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/55c6fb3d81f241d3abcebc3ed7254fee 2023-03-26 13:54:13,925 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/34016a20526c400d9cdff5c0ced61960 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/34016a20526c400d9cdff5c0ced61960 2023-03-26 13:54:13,930 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/34016a20526c400d9cdff5c0ced61960, entries=150, sequenceid=49, filesize=25.1 K 2023-03-26 13:54:13,931 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/13dd805b5c7347e0ab11e3ab5da9fc9c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/13dd805b5c7347e0ab11e3ab5da9fc9c 2023-03-26 13:54:13,937 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/13dd805b5c7347e0ab11e3ab5da9fc9c, entries=150, sequenceid=49, filesize=11.6 K 2023-03-26 13:54:13,938 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/55c6fb3d81f241d3abcebc3ed7254fee as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/55c6fb3d81f241d3abcebc3ed7254fee 2023-03-26 13:54:13,942 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/55c6fb3d81f241d3abcebc3ed7254fee, entries=150, sequenceid=49, filesize=11.6 K 2023-03-26 13:54:13,943 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~350.95 KB/359376, currentSize=67.09 KB/68700 for a739eafaf8b68b3135ee013236515a80 in 639ms, sequenceid=49, compaction requested=false 2023-03-26 13:54:13,943 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:13,943 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:13,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:13,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,945 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:13,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:13,946 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,946 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:13,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:13,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:13,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:13,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:13,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a6d2423[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:13,948 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:13,948 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,949 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:13,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:13,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:13,951 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,951 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:13,953 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:13,953 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:13,953 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:13,953 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:13,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:13,955 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-26 13:54:13,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:13,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:13,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:13,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:13,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:13,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:13,963 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:13,963 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:13,963 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:13,963 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:13,963 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:13,966 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:13,966 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,966 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:13,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:13,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:13,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:13,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:13,968 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:13,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:13,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:13,971 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:13,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:13,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,977 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:13,977 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,977 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:13,977 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:13,977 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:13,977 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:13,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:13,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:13,977 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:13,977 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,977 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:13,977 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:13,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:13,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:13,978 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:13,978 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:13,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:13,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:13,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:14,078 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:14,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:14,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:14,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:14,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:14,093 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:14,093 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:14,093 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:14,093 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:14,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:14,095 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-26 13:54:14,096 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,096 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,097 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:14,097 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:14,097 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:14,097 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:14,097 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:14,097 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:14,097 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:14,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:14,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:14,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,098 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,099 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:14,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,099 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:14,099 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:14,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:14,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:14,104 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:14,104 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:14,104 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:14,117 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:14,117 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:14,117 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:14,117 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:14,117 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:14,118 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:14,118 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:14,118 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:14,118 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,118 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:14,118 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:14,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838914112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838914115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838914121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838914121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,129 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,129 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,129 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:14,130 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:14,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20166b84[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:14,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:14,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,134 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,135 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:14,135 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:14,135 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:14,135 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,135 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:14,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:14,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:14,137 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:14,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:14,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:14,142 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:14,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:14,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:14,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:14,142 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:14,145 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,145 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:14,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,145 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:14,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,148 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:14,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:14,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:14,153 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:14,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:14,153 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:14,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:14,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:14,153 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:14,153 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,153 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:14,153 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,153 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,153 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,153 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,153 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,167 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326bf02f9b5488344b29cc7863f122f2ce2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326bf02f9b5488344b29cc7863f122f2ce2 2023-03-26 13:54:14,191 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=64, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/fd77731c5f924e6989a2daa22fcb96ac 2023-03-26 13:54:14,220 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838914220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838914221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838914224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838914224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,253 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:14,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:14,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:14,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:14,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:14,257 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:14,258 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:14,258 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:14,258 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:14,258 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,258 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,259 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:14,260 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:14,261 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:14,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:14,262 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:14,262 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:14,262 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:14,262 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:14,264 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:14,264 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:14,264 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:14,264 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:14,264 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:14,264 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:14,264 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:14,264 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:14,264 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,264 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:14,264 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:14,266 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:14,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:14,268 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:14,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a763aa2[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:14,268 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,270 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,270 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:14,270 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:14,270 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,270 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:14,270 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:14,272 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,272 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:14,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,272 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:14,272 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:14,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:14,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:14,275 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:14,275 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:14,275 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:14,275 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:14,275 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:14,277 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,277 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,277 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:14,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:14,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:14,277 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,278 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,281 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,281 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,281 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,281 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:14,281 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:14,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:14,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:14,281 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:14,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,281 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:14,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:14,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:14,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:14,281 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,282 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:14,282 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,282 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:14,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:14,282 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,382 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:14,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:14,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:14,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:14,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:14,388 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:14,388 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:14,388 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:14,388 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:14,389 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,389 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,390 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:14,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:14,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:14,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:14,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:14,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:14,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:14,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:14,392 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:14,392 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:14,392 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:14,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:14,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:14,396 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:14,396 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:14,396 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:14,396 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:14,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:14,397 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:14,397 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,397 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:14,397 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:14,398 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,399 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,399 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:14,399 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:14,401 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:14,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36285e80[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:14,401 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,403 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,403 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:14,403 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:14,403 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:14,404 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,404 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:14,407 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:14,407 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:14,407 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:14,407 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:14,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:14,411 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:14,412 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:14,412 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:14,412 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:14,412 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:14,414 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,414 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,414 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:14,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,416 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,418 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,421 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,422 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,422 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,422 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,422 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:14,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:14,422 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:14,422 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,422 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,422 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:14,422 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:14,422 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:14,422 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:14,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:14,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:14,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:14,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:14,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:14,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838914423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838914424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838914428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838914429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838914485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,522 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:14,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:14,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:14,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:14,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:14,528 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:14,528 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:14,528 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:14,528 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:14,529 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,529 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,530 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:14,530 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:14,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:14,531 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,531 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:14,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:14,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,531 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:14,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:14,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:14,532 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:14,532 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:14,532 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:14,532 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:14,532 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:14,532 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:14,532 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:14,532 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:14,532 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:14,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:14,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:14,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:14,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:14,538 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,538 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,538 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,538 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:14,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:14,540 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:14,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3567285a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:14,540 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,541 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,541 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:14,541 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:14,541 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:14,542 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,542 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:14,543 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:14,543 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,543 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,543 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:14,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:14,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:14,546 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:14,547 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:14,547 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:14,547 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:14,547 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:14,549 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,549 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,549 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,550 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,551 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:14,554 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:14,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:14,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:14,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:14,556 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,556 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,556 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,556 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,556 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:14,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:14,556 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,556 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,556 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:14,556 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:14,557 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:14,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:14,623 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/e73de165ec8f475da06efc9533e962e4 2023-03-26 13:54:14,641 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=64 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/63368c71628741af83a8929e0528f8b3 2023-03-26 13:54:14,647 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/fd77731c5f924e6989a2daa22fcb96ac as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/fd77731c5f924e6989a2daa22fcb96ac 2023-03-26 13:54:14,651 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/fd77731c5f924e6989a2daa22fcb96ac, entries=150, sequenceid=64, filesize=25.1 K 2023-03-26 13:54:14,652 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/e73de165ec8f475da06efc9533e962e4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/e73de165ec8f475da06efc9533e962e4 2023-03-26 13:54:14,657 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:14,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:14,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:14,660 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/e73de165ec8f475da06efc9533e962e4, entries=150, sequenceid=64, filesize=11.6 K 2023-03-26 13:54:14,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:14,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:14,661 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:14,661 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:14,661 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:14,661 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:14,661 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,662 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/63368c71628741af83a8929e0528f8b3 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/63368c71628741af83a8929e0528f8b3 2023-03-26 13:54:14,662 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,663 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,663 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:14,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:14,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:14,663 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,663 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:14,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:14,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:14,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:14,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:14,665 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:14,665 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:14,665 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:14,666 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:14,666 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:14,666 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:14,666 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:14,666 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:14,666 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:14,666 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:14,666 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:14,666 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,666 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:14,667 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:14,667 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/63368c71628741af83a8929e0528f8b3, entries=150, sequenceid=64, filesize=11.6 K 2023-03-26 13:54:14,668 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for a739eafaf8b68b3135ee013236515a80 in 573ms, sequenceid=64, compaction requested=true 2023-03-26 13:54:14,668 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:14,668 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:14,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:14,668 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:14,668 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:14,668 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,670 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77238 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:14,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,670 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/A is initiating minor compaction (all files) 2023-03-26 13:54:14,670 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/A in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:14,670 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/63dd65ae30f64f0d8ea160ea9b8f55b9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/34016a20526c400d9cdff5c0ced61960, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/fd77731c5f924e6989a2daa22fcb96ac] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=75.4 K 2023-03-26 13:54:14,671 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:14,671 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 63dd65ae30f64f0d8ea160ea9b8f55b9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679838852281 2023-03-26 13:54:14,671 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/B is initiating minor compaction (all files) 2023-03-26 13:54:14,672 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/B in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:14,672 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/4fb2e4ac1db9405c8f93f262744b933c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/13dd805b5c7347e0ab11e3ab5da9fc9c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/e73de165ec8f475da06efc9533e962e4] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=34.8 K 2023-03-26 13:54:14,672 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 34016a20526c400d9cdff5c0ced61960, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1679838852342 2023-03-26 13:54:14,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,672 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 4fb2e4ac1db9405c8f93f262744b933c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679838852281 2023-03-26 13:54:14,672 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting fd77731c5f924e6989a2daa22fcb96ac, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1679838853468 2023-03-26 13:54:14,673 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 13dd805b5c7347e0ab11e3ab5da9fc9c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1679838852342 2023-03-26 13:54:14,673 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e73de165ec8f475da06efc9533e962e4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1679838853468 2023-03-26 13:54:14,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:14,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791e310e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:14,677 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:14,677 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,681 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,681 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:14,681 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:14,681 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:14,681 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,681 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:14,682 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:14,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:14,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:14,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:14,685 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:14,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:14,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:14,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:14,686 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:14,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:14,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:14,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:14,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,693 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,696 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,700 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:14,700 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:14,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:14,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:14,700 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:14,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:14,700 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:14,700 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,701 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:14,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:14,701 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:14,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:14,703 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#B#compaction#145 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:14,715 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#A#compaction#144 average throughput is 0.56 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:14,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:14,731 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-26 13:54:14,731 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:14,731 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:14,731 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:14,731 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:14,731 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:14,731 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:14,743 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/8472c33a12c7483eb3f4f5d8efbd632e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/8472c33a12c7483eb3f4f5d8efbd632e 2023-03-26 13:54:14,749 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a739eafaf8b68b3135ee013236515a80/B of a739eafaf8b68b3135ee013236515a80 into 8472c33a12c7483eb3f4f5d8efbd632e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:14,749 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:14,749 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/B, priority=13, startTime=1679838854668; duration=0sec 2023-03-26 13:54:14,750 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:14,750 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:14,752 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:14,752 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/C is initiating minor compaction (all files) 2023-03-26 13:54:14,752 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/C in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:14,752 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/22e30b94be774f1092dbf9f2255e4cbb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/55c6fb3d81f241d3abcebc3ed7254fee, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/63368c71628741af83a8929e0528f8b3] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=34.8 K 2023-03-26 13:54:14,753 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 22e30b94be774f1092dbf9f2255e4cbb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679838852281 2023-03-26 13:54:14,754 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 55c6fb3d81f241d3abcebc3ed7254fee, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1679838852342 2023-03-26 13:54:14,755 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 63368c71628741af83a8929e0528f8b3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1679838853468 2023-03-26 13:54:14,758 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326687167ef77854fd3a0cc568e62632436 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326687167ef77854fd3a0cc568e62632436 2023-03-26 13:54:14,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838914752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,767 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=83, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/9b65cc6695a245e9908bf41d67d6eb48 2023-03-26 13:54:14,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838914763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838914764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,770 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#C#compaction#147 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:14,772 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838914769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,790 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/990c9016eee24195b526aa1debefcf0f 2023-03-26 13:54:14,801 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:14,801 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/172f8a97f0c54fdd809e3271c470dd80 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/172f8a97f0c54fdd809e3271c470dd80 2023-03-26 13:54:14,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:14,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:14,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:14,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:14,809 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:14,809 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:14,809 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:14,809 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:14,812 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a739eafaf8b68b3135ee013236515a80/C of a739eafaf8b68b3135ee013236515a80 into 172f8a97f0c54fdd809e3271c470dd80(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:14,812 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:14,812 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/C, priority=13, startTime=1679838854668; duration=0sec 2023-03-26 13:54:14,812 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:14,813 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,813 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,815 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:14,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,816 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,817 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,817 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:14,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:14,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:14,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:14,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:14,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:14,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:14,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:14,825 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:14,825 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:14,825 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:14,825 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:14,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:14,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:14,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:14,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:14,827 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,827 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,827 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,827 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:14,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7537e38f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:14,830 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:14,830 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,831 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,831 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:14,832 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,832 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:14,832 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:14,832 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:14,834 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:14,834 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:14,834 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,834 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:14,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:14,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:14,837 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:14,837 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:14,837 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:14,837 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:14,837 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:14,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:14,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:14,843 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,844 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,847 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:14,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:14,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:14,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:14,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:14,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:14,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:14,847 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:14,847 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:14,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:14,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:14,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838914863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838914870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838914873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:14,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838914873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,951 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:14,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:14,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:14,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:14,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:14,957 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:14,957 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:14,957 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:14,957 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:14,957 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,957 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,959 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:14,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:14,959 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:14,960 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,960 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:14,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:14,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:14,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:14,965 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:14,965 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:14,965 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:14,967 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:14,967 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:14,967 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:14,967 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:14,967 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:14,967 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:14,967 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:14,967 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:14,967 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,967 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:14,967 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,969 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:14,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:14,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22a7cdf7[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:14,971 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:14,971 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,976 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,976 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:14,976 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:14,976 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:14,976 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,976 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:14,978 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:14,978 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:14,978 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:14,978 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:14,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:14,983 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:14,983 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:14,983 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:14,983 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:14,983 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:14,986 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,986 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:14,986 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:14,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:14,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:14,987 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,990 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,992 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,992 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,992 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:14,992 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:14,992 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:14,992 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:14,992 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:14,993 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,993 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:14,993 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:14,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:14,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:14,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:14,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:14,996 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:14,996 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:15,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:15,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838915066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:15,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:15,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838915074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838915075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:15,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838915075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,096 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:15,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:15,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:15,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:15,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:15,100 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:15,100 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:15,100 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:15,100 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:15,101 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,101 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:15,103 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:15,103 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:15,103 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,103 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:15,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:15,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:15,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:15,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:15,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:15,104 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:15,104 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:15,104 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:15,105 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:15,105 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:15,105 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:15,105 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:15,105 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:15,105 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:15,105 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:15,105 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:15,105 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,105 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:15,105 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:15,107 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,107 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:15,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:15,107 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,107 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:15,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:15,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:15,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:15,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:15,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:15,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a3a6e47[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:15,109 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:15,109 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,110 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,110 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:15,110 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:15,110 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:15,110 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,110 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:15,111 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:15,111 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:15,111 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:15,111 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:15,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:15,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:15,114 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:15,114 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:15,114 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:15,114 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:15,114 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:15,116 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,116 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:15,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,116 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:15,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:15,117 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:15,118 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:15,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:15,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,124 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,124 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:15,124 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:15,124 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,124 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,124 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,124 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:15,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:15,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:15,124 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,124 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:15,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:15,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:15,125 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:15,125 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:15,125 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:15,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:15,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:15,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:15,221 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/06274ad312e34c2f85d1dfd9a144b15b 2023-03-26 13:54:15,225 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:15,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:15,226 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/9b65cc6695a245e9908bf41d67d6eb48 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/9b65cc6695a245e9908bf41d67d6eb48 2023-03-26 13:54:15,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:15,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:15,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:15,230 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:15,230 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:15,230 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:15,230 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:15,230 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/9b65cc6695a245e9908bf41d67d6eb48, entries=200, sequenceid=83, filesize=31.9 K 2023-03-26 13:54:15,230 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,230 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:15,232 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/990c9016eee24195b526aa1debefcf0f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/990c9016eee24195b526aa1debefcf0f 2023-03-26 13:54:15,233 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:15,233 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:15,233 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,234 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:15,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:15,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:15,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:15,235 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:15,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:15,235 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:15,235 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:15,235 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:15,235 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:15,235 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:15,235 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:15,235 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:15,235 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:15,235 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:15,235 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:15,235 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:15,235 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,235 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:15,235 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:15,237 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,237 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:15,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:15,237 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/990c9016eee24195b526aa1debefcf0f, entries=150, sequenceid=83, filesize=11.6 K 2023-03-26 13:54:15,237 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,237 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:15,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:15,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:15,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:15,238 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/06274ad312e34c2f85d1dfd9a144b15b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/06274ad312e34c2f85d1dfd9a144b15b 2023-03-26 13:54:15,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:15,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:15,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1224aa4f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:15,239 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:15,239 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,241 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,241 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:15,241 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:15,241 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,241 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:15,241 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:15,243 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,243 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:15,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:15,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:15,243 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:15,243 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:15,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:15,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:15,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:15,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:15,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:15,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:15,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:15,245 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:15,247 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:15,247 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:15,247 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:15,247 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:15,248 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,249 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:15,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,249 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:15,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:15,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:15,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:15,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:15,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,250 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:15,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:15,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:15,251 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:15,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:15,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,255 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:15,255 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:15,255 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:15,255 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:15,255 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:15,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:15,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:15,255 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:15,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:15,255 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,255 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,255 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,255 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,256 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:15,256 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:15,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:15,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:15,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:15,260 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/06274ad312e34c2f85d1dfd9a144b15b, entries=150, sequenceid=83, filesize=11.6 K 2023-03-26 13:54:15,261 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=80.51 KB/82440 for a739eafaf8b68b3135ee013236515a80 in 530ms, sequenceid=83, compaction requested=false 2023-03-26 13:54:15,261 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:15,356 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:15,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:15,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:15,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:15,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:15,360 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:15,360 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:15,360 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:15,360 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:15,360 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,360 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:15,364 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,364 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:15,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:15,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:15,365 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,365 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:15,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:15,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:15,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:15,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:15,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:15,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:15,366 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:15,366 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:15,366 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:15,366 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:15,366 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:15,366 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:15,366 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:15,366 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-26 13:54:15,367 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:15,367 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:15,367 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:15,367 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:15,367 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:15,367 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:15,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:15,372 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:15,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:15,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838915392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:15,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838915392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:15,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838915393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:15,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838915393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,405 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326f2e56ee592b74ad09394747c3a1d8712 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f2e56ee592b74ad09394747c3a1d8712 2023-03-26 13:54:15,421 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=100, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/20acdfc0348a41e5bb1345f88ce26633 2023-03-26 13:54:15,438 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/767ed78b783d41989d740fd1e8781969 2023-03-26 13:54:15,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:15,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838915498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:15,501 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:15,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838915498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838915499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:15,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838915499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,553 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/2fa5e34796e848a68253b58566e1c262 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/2fa5e34796e848a68253b58566e1c262 2023-03-26 13:54:15,559 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a739eafaf8b68b3135ee013236515a80/A of a739eafaf8b68b3135ee013236515a80 into 2fa5e34796e848a68253b58566e1c262(size=25.2 K), total size for store is 57.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:15,559 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:15,559 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/A, priority=13, startTime=1679838854668; duration=0sec 2023-03-26 13:54:15,559 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:15,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:15,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838915702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,702 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:15,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838915702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:15,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838915703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:15,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838915704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,862 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=100 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/4cfc9bb3ec624560a4dd36addacd78fc 2023-03-26 13:54:15,867 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/20acdfc0348a41e5bb1345f88ce26633 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/20acdfc0348a41e5bb1345f88ce26633 2023-03-26 13:54:15,872 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/20acdfc0348a41e5bb1345f88ce26633, entries=150, sequenceid=100, filesize=25.1 K 2023-03-26 13:54:15,873 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/767ed78b783d41989d740fd1e8781969 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/767ed78b783d41989d740fd1e8781969 2023-03-26 13:54:15,878 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/767ed78b783d41989d740fd1e8781969, entries=150, sequenceid=100, filesize=11.6 K 2023-03-26 13:54:15,879 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/4cfc9bb3ec624560a4dd36addacd78fc as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/4cfc9bb3ec624560a4dd36addacd78fc 2023-03-26 13:54:15,883 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/4cfc9bb3ec624560a4dd36addacd78fc, entries=150, sequenceid=100, filesize=11.6 K 2023-03-26 13:54:15,883 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for a739eafaf8b68b3135ee013236515a80 in 517ms, sequenceid=100, compaction requested=true 2023-03-26 13:54:15,883 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:15,883 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:15,884 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:15,884 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:15,884 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,884 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:15,884 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:15,885 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,885 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:15,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:15,886 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,886 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:15,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:15,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:15,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:15,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:15,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:15,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58c5181f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:15,887 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:15,887 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,888 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:15,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:15,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:15,889 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,889 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:15,890 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:15,890 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:15,890 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:15,890 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:15,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:15,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:15,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:15,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:15,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:15,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:15,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:15,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:15,893 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:15,893 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:15,893 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:15,893 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:15,893 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:15,903 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,903 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:15,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:15,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,903 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:15,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:15,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:15,903 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:15,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:15,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:15,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:15,905 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:15,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:15,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,907 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,907 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:15,907 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:15,907 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:15,907 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:15,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:15,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:15,915 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:15,915 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,915 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:15,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:15,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:15,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:15,915 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:15,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:15,915 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:15,916 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:15,916 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:16,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:16,007 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-26 13:54:16,007 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:16,008 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:16,008 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:16,008 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:16,008 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:16,008 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:16,016 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:16,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:16,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:16,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:16,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:16,021 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:16,021 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:16,021 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:16,021 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:16,022 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,022 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,023 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,023 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:16,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:16,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:16,023 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,023 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:16,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:16,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838916019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:16,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:16,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:16,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838916019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,025 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:16,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:16,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:16,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838916019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,025 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:16,025 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:16,025 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:16,025 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:16,025 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:16,025 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:16,025 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:16,025 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:16,025 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:16,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:16,026 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:16,026 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:16,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838916024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,026 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,026 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:16,026 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,027 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,027 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,028 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:16,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:16,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4411be85[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:16,031 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:16,031 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,033 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,034 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:16,034 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,034 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:16,034 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:16,034 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:16,035 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326af3b4b60a49648ecbbc03ed2f0c4de12 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326af3b4b60a49648ecbbc03ed2f0c4de12 2023-03-26 13:54:16,035 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:16,035 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:16,035 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:16,035 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:16,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:16,040 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:16,040 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:16,040 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:16,040 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:16,040 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:16,042 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,042 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:16,042 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,043 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=121, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/519a5a9e89134998affe1ad0fbd0c77f 2023-03-26 13:54:16,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:16,044 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:16,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,045 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,048 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,048 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,048 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,048 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,048 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:16,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:16,048 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:16,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:16,048 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:16,048 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:16,048 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,049 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,048 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:16,049 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:16,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:16,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:16,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:16,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:16,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838916125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:16,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838916126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:16,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838916126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:16,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838916127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,148 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:16,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:16,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:16,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:16,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:16,155 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:16,155 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:16,155 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:16,155 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:16,155 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,159 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,159 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:16,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:16,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:16,160 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,160 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:16,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:16,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:16,161 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:16,161 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:16,161 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:16,161 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:16,161 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:16,161 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:16,161 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:16,161 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:16,161 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:16,162 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:16,162 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:16,162 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:16,162 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:16,162 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,162 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:16,162 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:16,163 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,163 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,164 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,164 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:16,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d10b816[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:16,166 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:16,166 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,167 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,167 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:16,167 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,167 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:16,167 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:16,167 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:16,169 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,169 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:16,169 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:16,169 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:16,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:16,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:16,173 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:16,174 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:16,174 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:16,174 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:16,174 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:16,181 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,181 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,181 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:16,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:16,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:16,182 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,186 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,188 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:16,189 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:16,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:16,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:16,188 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:16,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:16,189 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:16,189 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,189 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,189 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,189 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,189 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,189 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,189 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:16,189 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:16,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:16,289 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:16,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:16,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:16,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:16,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:16,293 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:16,293 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:16,293 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:16,293 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:16,294 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,294 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:16,295 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:16,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:16,295 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,295 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:16,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:16,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:16,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:16,297 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:16,297 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:16,297 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:16,297 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:16,297 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:16,297 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:16,297 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:16,297 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:16,297 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:16,297 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:16,297 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:16,297 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,297 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:16,297 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:16,299 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,299 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,299 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,299 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:16,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:16,301 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:16,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a510d29[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:16,301 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,303 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,304 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,304 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:16,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:16,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:16,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:16,305 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,305 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:16,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,305 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:16,305 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:16,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:16,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:16,308 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:16,308 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:16,308 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:16,308 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:16,308 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:16,310 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,310 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,310 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,312 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,319 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,319 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:16,319 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:16,319 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,319 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:16,319 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:16,319 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,319 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,319 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:16,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:16,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:16,319 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:16,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:16,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:16,319 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:16,320 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:16,328 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:16,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838916328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:16,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838916328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:16,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838916328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,331 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:16,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838916330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,420 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:16,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:16,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:16,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:16,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:16,423 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:16,423 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:16,423 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:16,423 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:16,425 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,425 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,426 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,426 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:16,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:16,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:16,427 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,427 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:16,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:16,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:16,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:16,428 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:16,428 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:16,428 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:16,428 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:16,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:16,430 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,430 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,430 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,430 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:16,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:16,432 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:16,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aa4ff9c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:16,432 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,433 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,433 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:16,433 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:16,433 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:16,433 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,433 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:16,436 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,436 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:16,436 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:16,436 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:16,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:16,438 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:16,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:16,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:16,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:16,438 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:16,440 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,440 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,440 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,441 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:16,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:16,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,442 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,444 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,444 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,444 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:16,444 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,444 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:16,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:16,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:16,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:16,444 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,444 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,444 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,444 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,444 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:16,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:16,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:16,445 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:16,445 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:16,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:16,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:16,461 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/f95f75f560da416a8a692087006d7ae0 2023-03-26 13:54:16,510 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:16,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838916508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,512 DEBUG [Thread-1035] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., hostname=jenkins-hbase4.apache.org,42517,1679838814721, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:54:16,545 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:16,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:16,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:16,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:16,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:16,548 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:16,548 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:16,548 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:16,548 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:16,549 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,549 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,551 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:16,551 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:16,551 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,551 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:16,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:16,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:16,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:16,552 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:16,552 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:16,552 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:16,553 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:16,553 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:16,553 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:16,553 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:16,553 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:16,553 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:16,553 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:16,553 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:16,553 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,553 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:16,553 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:16,556 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,556 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,556 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,556 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:16,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:16,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30885db[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:16,558 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:16,558 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,559 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,559 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:16,559 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:16,559 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:16,559 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,559 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:16,560 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:16,560 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:16,560 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,560 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:16,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:16,564 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:16,564 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:16,564 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:16,564 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:16,564 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:16,565 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,565 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,565 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:16,566 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,567 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,570 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:16,570 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:16,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:16,570 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,570 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:16,570 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:16,570 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:16,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:16,570 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:16,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:16,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:16,570 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,571 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,571 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,571 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:16,571 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:16,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:16,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838916629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:16,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838916631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,632 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:16,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838916631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:16,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838916634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,671 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:16,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:16,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:16,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:16,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:16,674 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:16,675 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:16,675 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:16,675 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:16,675 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,675 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,676 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,676 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:16,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:16,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:16,676 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,676 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:16,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:16,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:16,678 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:16,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:16,678 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:16,678 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:16,678 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:16,680 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:16,680 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:16,680 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:16,680 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:16,682 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:16,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:16,684 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:16,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bd05da4[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:16,684 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,685 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,685 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:16,685 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:16,686 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:16,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:16,688 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,688 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:16,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,688 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:16,689 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:16,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:16,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:16,691 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:16,691 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:16,691 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:16,691 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:16,691 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:16,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:16,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,693 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,694 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,699 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:16,699 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:16,699 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:16,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:16,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:16,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:16,699 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:16,699 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:16,699 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:16,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:16,699 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,700 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,700 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,700 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,700 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,700 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:16,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:16,800 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:16,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:16,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:16,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:16,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:16,803 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:16,803 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:16,804 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:16,804 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:16,804 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,804 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,808 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:16,808 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:16,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:16,809 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,809 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:16,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:16,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:16,810 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:16,810 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:16,810 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:16,810 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:16,810 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:16,810 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:16,810 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:16,810 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:16,810 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:16,810 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:16,810 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:16,810 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:16,811 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:16,811 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,811 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:16,811 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:16,812 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,812 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,812 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,812 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:16,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39da21b2[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:16,814 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:16,814 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,815 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:16,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:16,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:16,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:16,816 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,816 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:16,817 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:16,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,817 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:16,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:16,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:16,819 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:16,819 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:16,819 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:16,819 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:16,819 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:16,821 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,821 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:16,821 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:16,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:16,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:16,822 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:16,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:16,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:16,823 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:16,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:16,825 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,825 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,825 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,825 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,825 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,825 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:16,825 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:16,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:16,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:16,825 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:16,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:16,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:16,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:16,825 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:16,825 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,826 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:16,826 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:16,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:16,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:16,874 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/8bd94a0333754397817cc8b378ea3c56 2023-03-26 13:54:16,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/519a5a9e89134998affe1ad0fbd0c77f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/519a5a9e89134998affe1ad0fbd0c77f 2023-03-26 13:54:16,883 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/519a5a9e89134998affe1ad0fbd0c77f, entries=250, sequenceid=121, filesize=38.7 K 2023-03-26 13:54:16,884 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/f95f75f560da416a8a692087006d7ae0 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/f95f75f560da416a8a692087006d7ae0 2023-03-26 13:54:16,888 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/f95f75f560da416a8a692087006d7ae0, entries=150, sequenceid=121, filesize=11.6 K 2023-03-26 13:54:16,888 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/8bd94a0333754397817cc8b378ea3c56 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/8bd94a0333754397817cc8b378ea3c56 2023-03-26 13:54:16,893 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/8bd94a0333754397817cc8b378ea3c56, entries=150, sequenceid=121, filesize=11.6 K 2023-03-26 13:54:16,893 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a739eafaf8b68b3135ee013236515a80 in 886ms, sequenceid=121, compaction requested=true 2023-03-26 13:54:16,894 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:16,894 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:16,894 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:54:16,894 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:54:16,894 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:16,894 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:16,895 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 123911 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:54:16,895 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/A is initiating minor compaction (all files) 2023-03-26 13:54:16,895 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/A in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:16,895 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:54:16,895 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/2fa5e34796e848a68253b58566e1c262, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/9b65cc6695a245e9908bf41d67d6eb48, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/20acdfc0348a41e5bb1345f88ce26633, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/519a5a9e89134998affe1ad0fbd0c77f] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=121.0 K 2023-03-26 13:54:16,895 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/B is initiating minor compaction (all files) 2023-03-26 13:54:16,895 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/B in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:16,895 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/8472c33a12c7483eb3f4f5d8efbd632e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/990c9016eee24195b526aa1debefcf0f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/767ed78b783d41989d740fd1e8781969, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/f95f75f560da416a8a692087006d7ae0] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=46.5 K 2023-03-26 13:54:16,896 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 2fa5e34796e848a68253b58566e1c262, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1679838853468 2023-03-26 13:54:16,896 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 8472c33a12c7483eb3f4f5d8efbd632e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1679838853468 2023-03-26 13:54:16,896 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 9b65cc6695a245e9908bf41d67d6eb48, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679838854110 2023-03-26 13:54:16,897 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 990c9016eee24195b526aa1debefcf0f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679838854110 2023-03-26 13:54:16,899 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 20acdfc0348a41e5bb1345f88ce26633, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1679838854755 2023-03-26 13:54:16,899 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 767ed78b783d41989d740fd1e8781969, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1679838854755 2023-03-26 13:54:16,900 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 519a5a9e89134998affe1ad0fbd0c77f, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679838855389 2023-03-26 13:54:16,900 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting f95f75f560da416a8a692087006d7ae0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679838855389 2023-03-26 13:54:16,911 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#B#compaction#157 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:16,917 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#A#compaction#156 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:16,926 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:16,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:16,927 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/06dfd74305b34ec8aa19275ecfe7bae6 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/06dfd74305b34ec8aa19275ecfe7bae6 2023-03-26 13:54:16,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:16,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:16,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:16,930 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:16,930 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:16,930 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:16,930 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:16,931 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,931 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,932 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:16,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:16,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:16,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:16,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:16,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:16,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:16,934 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:16,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:16,934 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:16,934 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:16,934 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:16,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:16,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:16,935 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:16,935 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:16,935 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a739eafaf8b68b3135ee013236515a80/B of a739eafaf8b68b3135ee013236515a80 into 06dfd74305b34ec8aa19275ecfe7bae6(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:16,935 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:16,935 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 13:54:16,935 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/B, priority=12, startTime=1679838856894; duration=0sec 2023-03-26 13:54:16,935 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:16,935 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:16,935 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:16,935 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:16,935 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:54:16,936 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:16,936 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:16,936 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:16,936 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:16,936 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:16,937 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47647 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:54:16,937 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/C is initiating minor compaction (all files) 2023-03-26 13:54:16,937 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/C in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:16,938 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/172f8a97f0c54fdd809e3271c470dd80, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/06274ad312e34c2f85d1dfd9a144b15b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/4cfc9bb3ec624560a4dd36addacd78fc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/8bd94a0333754397817cc8b378ea3c56] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=46.5 K 2023-03-26 13:54:16,938 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 172f8a97f0c54fdd809e3271c470dd80, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=64, earliestPutTs=1679838853468 2023-03-26 13:54:16,938 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 06274ad312e34c2f85d1dfd9a144b15b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1679838854110 2023-03-26 13:54:16,939 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 4cfc9bb3ec624560a4dd36addacd78fc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=100, earliestPutTs=1679838854755 2023-03-26 13:54:16,940 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 8bd94a0333754397817cc8b378ea3c56, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679838855389 2023-03-26 13:54:16,947 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/ab79e1d726ad4b4f936db947a46d6c71 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/ab79e1d726ad4b4f936db947a46d6c71 2023-03-26 13:54:16,951 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#C#compaction#159 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:16,961 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a739eafaf8b68b3135ee013236515a80/A of a739eafaf8b68b3135ee013236515a80 into ab79e1d726ad4b4f936db947a46d6c71(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:16,962 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:16,962 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/A, priority=12, startTime=1679838856894; duration=0sec 2023-03-26 13:54:16,962 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:16,965 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326383022ad871343e193b19c1e5fc84297 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326383022ad871343e193b19c1e5fc84297 2023-03-26 13:54:16,971 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=134, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/44e1d917e3ee4319a1a5b813e81ccbd9 2023-03-26 13:54:16,972 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/a0ed65be43e446768509dbf67f85237a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/a0ed65be43e446768509dbf67f85237a 2023-03-26 13:54:16,979 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a739eafaf8b68b3135ee013236515a80/C of a739eafaf8b68b3135ee013236515a80 into a0ed65be43e446768509dbf67f85237a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:16,980 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:16,980 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/C, priority=12, startTime=1679838856894; duration=0sec 2023-03-26 13:54:16,980 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:17,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:17,135 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:17,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:17,185 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:17,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838917179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838857176, max timestamp=1679838857180Num uniques 859; 2023-03-26 13:54:17,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:54:17,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838917178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838857176, max timestamp=1679838857180Num uniques 859; 2023-03-26 13:54:17,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:54:17,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679838857176, max timestamp=1679838857180Num uniques 859; 2023-03-26 13:54:17,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:54:17,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:17,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838917209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:17,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838917209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:17,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838917293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:17,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838917293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:17,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838917310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:17,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838917311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,388 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/89ab3dee95ce4b65a14a1d6b6db35edd 2023-03-26 13:54:17,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:17,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838917496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:17,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838917497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:17,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838917515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:17,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838917515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,800 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:17,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838917798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,801 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=134 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/a73bcb7d8c2742788f97bf9b8d310e7f 2023-03-26 13:54:17,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:17,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838917801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,806 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/44e1d917e3ee4319a1a5b813e81ccbd9 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/44e1d917e3ee4319a1a5b813e81ccbd9 2023-03-26 13:54:17,813 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/44e1d917e3ee4319a1a5b813e81ccbd9, entries=150, sequenceid=134, filesize=25.2 K 2023-03-26 13:54:17,813 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/89ab3dee95ce4b65a14a1d6b6db35edd as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/89ab3dee95ce4b65a14a1d6b6db35edd 2023-03-26 13:54:17,819 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/89ab3dee95ce4b65a14a1d6b6db35edd, entries=150, sequenceid=134, filesize=11.7 K 2023-03-26 13:54:17,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:17,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838917817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:17,820 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/a73bcb7d8c2742788f97bf9b8d310e7f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/a73bcb7d8c2742788f97bf9b8d310e7f 2023-03-26 13:54:17,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838917820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,829 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/a73bcb7d8c2742788f97bf9b8d310e7f, entries=150, sequenceid=134, filesize=11.7 K 2023-03-26 13:54:17,830 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for a739eafaf8b68b3135ee013236515a80 in 895ms, sequenceid=134, compaction requested=false 2023-03-26 13:54:17,830 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:17,830 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:17,830 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:17,831 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:17,831 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,831 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:17,831 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:17,832 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:17,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:17,832 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:17,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:17,833 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:17,833 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:17,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:17,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:17,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:17,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:17,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54ef610a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:17,834 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:17,834 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:17,836 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:17,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:17,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:17,836 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:17,836 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:17,836 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:17,836 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,836 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:17,837 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,837 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:17,838 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:17,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:17,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:17,838 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:17,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:17,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:17,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:17,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:17,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:17,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:17,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:17,840 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:17,840 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:17,840 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:17,840 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:17,840 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:17,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:17,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:17,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:17,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:17,843 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:17,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:17,844 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:17,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:17,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,846 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,846 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:17,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:17,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:17,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:17,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:17,846 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:17,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:17,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:17,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:17,847 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:17,847 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:17,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:17,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:17,847 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:17,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:17,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:17,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:17,947 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:17,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:17,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:17,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:17,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:17,951 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:17,952 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:17,952 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:17,952 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:17,952 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:17,952 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:17,953 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:17,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:17,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:17,954 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:17,954 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:17,954 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:17,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:17,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:17,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:17,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:17,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:17,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:17,957 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:17,957 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:17,957 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:17,957 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:17,957 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:17,957 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:17,957 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:17,957 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-03-26 13:54:17,957 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:17,957 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:17,957 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:17,958 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:17,958 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:17,958 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:17,983 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326d1fff24f1d294f6a9428105c2cb8af80 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326d1fff24f1d294f6a9428105c2cb8af80 2023-03-26 13:54:17,994 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=170, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/33b03d23e9f84ca8990d4f865acab751 2023-03-26 13:54:18,008 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/44b1fab7bb964146881fcc5c9c40bd12 2023-03-26 13:54:18,019 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/c29f7f7247e348aba502696491bc1e67 2023-03-26 13:54:18,023 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/33b03d23e9f84ca8990d4f865acab751 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/33b03d23e9f84ca8990d4f865acab751 2023-03-26 13:54:18,028 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/33b03d23e9f84ca8990d4f865acab751, entries=150, sequenceid=170, filesize=25.3 K 2023-03-26 13:54:18,029 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/44b1fab7bb964146881fcc5c9c40bd12 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/44b1fab7bb964146881fcc5c9c40bd12 2023-03-26 13:54:18,033 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/44b1fab7bb964146881fcc5c9c40bd12, entries=150, sequenceid=170, filesize=11.8 K 2023-03-26 13:54:18,033 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/c29f7f7247e348aba502696491bc1e67 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/c29f7f7247e348aba502696491bc1e67 2023-03-26 13:54:18,038 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/c29f7f7247e348aba502696491bc1e67, entries=150, sequenceid=170, filesize=11.8 K 2023-03-26 13:54:18,039 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=0 B/0 for a739eafaf8b68b3135ee013236515a80 in 82ms, sequenceid=170, compaction requested=true 2023-03-26 13:54:18,039 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:18,039 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:18,039 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:18,039 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:18,039 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,039 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:18,039 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:18,041 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:18,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:18,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@138e1011[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:18,043 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:18,043 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,045 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,045 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:18,045 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:18,045 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:18,046 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,046 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:18,047 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:18,047 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,047 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:18,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,047 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:18,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:18,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:18,050 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:18,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:18,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:18,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:18,050 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:18,051 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,051 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:18,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,051 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:18,052 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,054 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,056 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,056 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,056 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:18,056 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:18,057 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:18,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:18,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:18,056 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:18,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:18,057 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,057 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:18,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:18,057 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,057 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:18,057 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:18,057 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:18,158 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:18,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:18,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:18,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:18,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:18,162 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:18,162 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:18,162 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:18,162 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:18,162 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,162 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,165 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,165 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:18,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:18,165 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,165 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:18,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:18,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:18,167 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:18,167 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:18,167 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:18,167 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:18,167 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:18,167 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:18,167 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:18,167 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:18,168 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:18,168 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:18,168 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:18,168 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,168 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:18,168 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:18,172 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,172 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,173 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,173 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:18,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a3a52b9[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:18,175 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:18,175 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,177 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,178 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:18,178 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:18,178 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:18,178 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,178 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:18,181 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,181 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:18,181 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:18,181 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:18,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:18,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:18,184 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:18,184 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:18,184 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:18,184 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:18,184 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:18,186 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,186 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,186 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:18,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:18,186 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,192 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,195 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:18,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:18,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:18,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:18,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:18,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,196 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,196 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:18,196 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,196 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:18,196 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:18,196 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:18,196 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:18,196 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,196 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:18,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:18,296 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:18,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:18,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:18,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:18,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:18,304 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:18,304 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:18,304 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:18,304 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:18,304 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,304 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,306 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:18,306 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:18,306 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,306 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:18,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:18,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:18,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:18,308 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:18,308 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:18,308 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:18,309 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:18,309 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:18,309 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:18,309 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:18,310 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-03-26 13:54:18,310 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:18,310 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:18,310 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:18,310 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:18,311 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:18,311 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:18,348 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032603f36650b3764a04a5afa99a5f4011f5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032603f36650b3764a04a5afa99a5f4011f5 2023-03-26 13:54:18,362 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=175, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/511d71e8beab47e2972c60327aa36174 2023-03-26 13:54:18,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:18,367 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:18,381 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/a0027c460d73498f8ff22c201e2c0097 2023-03-26 13:54:18,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=559, cellSize=25595, totalHeapSize=58376, min timestamp=1679838858367, max timestamp=1679838858393Num uniques 859; 2023-03-26 13:54:18,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=25595, suffix heap size=117064,new segment heap size=58376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=559 2023-03-26 13:54:18,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=559, cellSize=25595, totalHeapSize=58376, min timestamp=1679838858367, max timestamp=1679838858393Num uniques 859; 2023-03-26 13:54:18,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=559, cellSize=25595, totalHeapSize=58376, min timestamp=1679838858367, max timestamp=1679838858393Num uniques 859; 2023-03-26 13:54:18,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=25595, suffix heap size=117064,new segment heap size=58376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=559 2023-03-26 13:54:18,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=25595, suffix heap size=117064,new segment heap size=58376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=559 2023-03-26 13:54:18,402 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/f4cf4bfc80b44a16af435661fcf060b9 2023-03-26 13:54:18,407 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/511d71e8beab47e2972c60327aa36174 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/511d71e8beab47e2972c60327aa36174 2023-03-26 13:54:18,414 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/511d71e8beab47e2972c60327aa36174, entries=50, sequenceid=175, filesize=11.6 K 2023-03-26 13:54:18,415 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/a0027c460d73498f8ff22c201e2c0097 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a0027c460d73498f8ff22c201e2c0097 2023-03-26 13:54:18,420 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a0027c460d73498f8ff22c201e2c0097, entries=50, sequenceid=175, filesize=7.1 K 2023-03-26 13:54:18,421 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/f4cf4bfc80b44a16af435661fcf060b9 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/f4cf4bfc80b44a16af435661fcf060b9 2023-03-26 13:54:18,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:18,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838918422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,427 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/f4cf4bfc80b44a16af435661fcf060b9, entries=50, sequenceid=175, filesize=7.1 K 2023-03-26 13:54:18,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:18,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838918424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,428 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=187.85 KB/192360 for a739eafaf8b68b3135ee013236515a80 in 119ms, sequenceid=175, compaction requested=true 2023-03-26 13:54:18,428 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:18,428 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:18,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:18,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:18,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,429 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:18,429 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:18,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:18,429 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=194.56 KB heapSize=527.48 KB 2023-03-26 13:54:18,430 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:18,430 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-26 13:54:18,430 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:18,430 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-26 13:54:18,430 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:18,430 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-26 13:54:18,431 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,431 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,432 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:18,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@720527fd[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:18,433 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:18,433 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,434 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,435 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:18,435 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:18,435 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,435 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:18,435 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:18,439 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:18,439 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:18,439 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:18,440 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:18,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:18,443 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:18,443 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:18,443 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:18,443 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:18,443 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:18,444 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,444 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,444 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:18,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:18,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:18,445 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,447 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:18,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838918429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:18,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838918431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,451 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:18,451 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:18,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:18,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:18,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:18,451 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:18,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:18,451 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326f2ce51ccc13b47aa9079653cef9f976d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f2ce51ccc13b47aa9079653cef9f976d 2023-03-26 13:54:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:18,452 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:18,452 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:18,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:18,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:18,453 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,453 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,453 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,453 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,453 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,453 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,453 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:18,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838918528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:18,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838918529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:18,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838918550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:18,552 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:18,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838918551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:18,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:18,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:18,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:18,556 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:18,556 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:18,556 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:18,556 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:18,556 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,556 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,558 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,558 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:18,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:18,558 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,558 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:18,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:18,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:18,559 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:18,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:18,559 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:18,559 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:18,559 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:18,559 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:18,559 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:18,559 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:18,559 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:18,559 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:18,560 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:18,560 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:18,561 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,561 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,562 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,562 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:18,563 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:18,563 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dcdff86[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:18,564 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,565 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,565 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:18,565 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:18,565 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:18,565 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,565 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:18,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,567 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:18,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,568 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:18,568 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:18,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:18,570 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:18,570 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:18,570 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:18,570 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:18,570 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:18,572 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,572 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:18,572 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,572 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,573 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,576 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,576 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,576 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,576 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,576 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:18,576 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:18,576 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:18,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:18,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:18,576 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,576 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,576 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:18,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:18,576 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:18,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:18,576 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:18,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:18,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:18,676 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:18,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:18,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:18,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:18,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:18,680 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:18,680 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:18,680 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:18,680 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:18,681 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,681 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,682 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:18,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:18,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,683 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:18,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:18,684 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:18,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:18,684 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:18,684 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:18,684 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:18,685 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,685 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:18,688 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:18,688 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:18,688 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:18,688 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:18,688 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:18,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,692 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,692 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,692 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:18,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ea062b6[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:18,694 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:18,694 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,699 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,699 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:18,699 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:18,699 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,699 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:18,699 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:18,700 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:18,700 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:18,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:18,704 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:18,704 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:18,704 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:18,704 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:18,704 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:18,706 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,706 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:18,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,706 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,707 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,709 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,711 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:18,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:18,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:18,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:18,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:18,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:18,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,711 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:18,712 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:18,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,712 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,712 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:18,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:18,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:18,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838918731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,733 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:18,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838918732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:18,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838918754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:18,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838918754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,812 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:18,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:18,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:18,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:18,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:18,815 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:18,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:18,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:18,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:18,816 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,816 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,817 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:18,817 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:18,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:18,817 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,818 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:18,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:18,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:18,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:18,819 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:18,819 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:18,819 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:18,819 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:18,819 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:18,820 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:18,820 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:18,820 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:18,820 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:18,820 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:18,820 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:18,820 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,821 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:18,821 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:18,822 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,822 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,822 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:18,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:18,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76475041[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:18,824 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:18,824 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,826 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:18,827 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,827 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:18,828 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,828 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:18,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,828 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:18,828 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:18,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:18,830 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:18,830 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:18,830 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:18,830 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:18,830 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:18,834 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,834 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,834 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:18,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:18,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,834 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,836 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,839 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:18,839 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:18,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:18,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:18,839 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,840 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,840 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,840 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,840 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,840 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,840 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,840 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:18,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:18,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:18,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:18,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:18,840 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:18,840 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:18,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:18,862 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=214, memsize=67.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/f63e6a8e0b1148f9936084c736ba6653 2023-03-26 13:54:18,940 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:18,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:18,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:18,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:18,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:18,944 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:18,944 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:18,944 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:18,944 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:18,944 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,944 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,947 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:18,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:18,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:18,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:18,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:18,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:18,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:18,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:18,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:18,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:18,950 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:18,951 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:18,951 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:18,951 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:18,951 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:18,951 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:18,951 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:18,952 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:18,952 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:18,952 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,952 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:18,952 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:18,953 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,953 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,956 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,956 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:18,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:18,957 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:18,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d26abbb[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:18,958 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,960 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:18,960 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:18,960 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,960 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:18,961 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,961 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:18,961 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,961 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:18,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:18,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:18,964 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:18,964 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:18,964 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:18,964 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:18,964 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:18,967 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,967 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:18,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,967 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:18,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:18,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:18,967 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:18,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:18,969 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:18,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:18,971 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:18,971 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:18,971 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:18,971 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:18,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:18,971 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:18,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:18,971 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:18,971 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:18,971 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,972 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:18,972 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:18,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:18,972 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:18,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:18,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:18,972 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:19,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:19,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838919034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:19,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838919037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:19,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838919060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:19,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838919060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,072 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:19,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:19,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:19,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:19,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:19,075 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:19,075 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:19,076 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:19,076 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:19,076 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,076 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,077 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:19,077 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:19,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:19,078 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,078 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:19,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:19,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:19,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:19,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:19,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:19,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:19,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:19,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:19,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:19,079 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:19,079 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:19,079 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:19,079 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:19,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:19,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:19,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:19,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:19,081 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,081 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,082 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,082 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:19,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:19,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f255056[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:19,083 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:19,083 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,085 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:19,085 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:19,085 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:19,086 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,086 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:19,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,089 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:19,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,089 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:19,089 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:19,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:19,092 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:19,092 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:19,092 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:19,092 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:19,092 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:19,093 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,093 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,093 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:19,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:19,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,094 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,095 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,099 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,099 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,099 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,099 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,099 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,099 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,100 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,100 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:19,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:19,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:19,100 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:19,100 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:19,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:19,100 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:19,100 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:19,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:19,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:19,200 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:19,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:19,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:19,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:19,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:19,204 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:19,204 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:19,204 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:19,205 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:19,205 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,205 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,206 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,206 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:19,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:19,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:19,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,207 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,207 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:19,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:19,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:19,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:19,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:19,208 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:19,208 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:19,208 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:19,208 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:19,208 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:19,208 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:19,208 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:19,209 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:19,209 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:19,209 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:19,209 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:19,209 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,209 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:19,209 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:19,210 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,210 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,210 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,211 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:19,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:19,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22ad34d6[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:19,213 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:19,213 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:19,214 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:19,214 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,214 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:19,218 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,218 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:19,218 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:19,218 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:19,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:19,222 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:19,222 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:19,222 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:19,222 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:19,222 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:19,223 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,223 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,223 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:19,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:19,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,224 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,225 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,237 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,237 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:19,237 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:19,237 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,237 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,237 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:19,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:19,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:19,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:19,238 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:19,238 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:19,237 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,238 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,238 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,238 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:19,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:19,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:19,293 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=67.09 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/96127f7d438a474794cb3d6b65892a7c 2023-03-26 13:54:19,338 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:19,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:19,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:19,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:19,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:19,344 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:19,344 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:19,344 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:19,344 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:19,344 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,345 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,347 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:19,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:19,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:19,347 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,347 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,347 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:19,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:19,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:19,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:19,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:19,348 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:19,348 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:19,348 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:19,352 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:19,352 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:19,352 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:19,352 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:19,352 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:19,352 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:19,352 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:19,352 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:19,352 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,352 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:19,352 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:19,355 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,355 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,355 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,355 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:19,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:19,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e24ad9a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:19,357 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:19,357 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,359 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,359 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:19,359 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:19,359 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:19,359 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,359 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:19,361 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:19,361 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:19,361 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:19,361 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:19,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:19,365 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:19,365 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:19,365 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:19,365 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:19,365 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:19,366 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,366 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,366 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:19,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:19,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:19,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,367 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,370 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,375 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:19,375 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:19,375 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,375 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,375 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:19,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:19,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:19,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:19,375 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:19,375 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,375 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:19,376 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:19,376 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,376 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:19,376 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:19,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:19,476 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:19,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:19,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:19,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:19,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:19,481 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:19,481 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:19,481 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:19,481 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:19,481 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,481 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,483 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:19,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:19,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,483 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:19,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:19,484 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,484 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:19,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:19,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:19,484 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:19,484 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:19,484 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:19,484 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:19,484 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:19,484 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:19,484 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:19,485 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:19,485 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:19,485 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:19,485 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:19,485 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,485 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:19,485 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:19,486 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,486 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,487 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,487 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:19,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:19,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7647a896[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:19,488 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:19,488 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,489 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,489 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:19,490 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:19,490 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:19,490 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,490 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:19,492 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:19,492 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:19,492 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,492 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:19,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:19,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:19,496 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:19,496 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:19,496 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:19,496 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:19,496 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:19,497 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,497 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,497 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:19,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,498 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,499 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:19,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:19,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:19,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:19,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:19,505 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:19,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,506 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,506 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:19,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:19,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:19,506 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:19,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:19,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:19,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:19,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838919539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:19,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838919543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:19,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838919563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:19,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838919565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,606 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:19,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:19,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:19,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:19,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:19,612 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:19,612 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:19,612 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:19,612 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:19,613 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,613 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,614 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:19,614 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:19,615 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,615 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:19,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:19,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:19,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:19,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:19,617 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:19,617 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:19,617 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:19,617 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:19,617 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:19,617 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:19,617 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:19,617 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:19,617 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:19,617 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:19,617 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:19,617 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,617 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:19,617 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:19,619 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,619 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,619 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:19,619 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:19,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43d5e1e8[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:19,621 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:19,621 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,623 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,623 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:19,623 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:19,623 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:19,623 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,623 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:19,625 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:19,625 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:19,625 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:19,625 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:19,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:19,628 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:19,628 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:19,628 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:19,628 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:19,628 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:19,631 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,631 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:19,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,631 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:19,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:19,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,633 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,635 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,638 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:19,638 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:19,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:19,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:19,638 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:19,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:19,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:19,638 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:19,638 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:19,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:19,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:19,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:19,640 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,640 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,640 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,640 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,640 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,640 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,640 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,717 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=67.09 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/f5d5b77dc9fe4693b7e834e736d83083 2023-03-26 13:54:19,721 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/f63e6a8e0b1148f9936084c736ba6653 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/f63e6a8e0b1148f9936084c736ba6653 2023-03-26 13:54:19,726 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/f63e6a8e0b1148f9936084c736ba6653, entries=250, sequenceid=214, filesize=38.9 K 2023-03-26 13:54:19,727 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/96127f7d438a474794cb3d6b65892a7c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/96127f7d438a474794cb3d6b65892a7c 2023-03-26 13:54:19,731 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/96127f7d438a474794cb3d6b65892a7c, entries=150, sequenceid=214, filesize=11.8 K 2023-03-26 13:54:19,732 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/f5d5b77dc9fe4693b7e834e736d83083 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/f5d5b77dc9fe4693b7e834e736d83083 2023-03-26 13:54:19,737 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/f5d5b77dc9fe4693b7e834e736d83083, entries=150, sequenceid=214, filesize=11.8 K 2023-03-26 13:54:19,738 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~201.27 KB/206100, heapSize ~547.36 KB/560496, currentSize=0 B/0 for a739eafaf8b68b3135ee013236515a80 in 1309ms, sequenceid=214, compaction requested=true 2023-03-26 13:54:19,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:19,738 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 13:54:19,738 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:19,739 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:19,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:19,740 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 13:54:19,740 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:19,741 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:19,741 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 129514 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 13:54:19,741 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/A is initiating minor compaction (all files) 2023-03-26 13:54:19,741 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/A in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:19,741 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/ab79e1d726ad4b4f936db947a46d6c71, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/44e1d917e3ee4319a1a5b813e81ccbd9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/33b03d23e9f84ca8990d4f865acab751, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/511d71e8beab47e2972c60327aa36174, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/f63e6a8e0b1148f9936084c736ba6653] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=126.5 K 2023-03-26 13:54:19,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:19,742 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting ab79e1d726ad4b4f936db947a46d6c71, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679838855389 2023-03-26 13:54:19,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:19,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:19,743 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:19,743 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:19,743 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:19,743 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:19,743 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 44e1d917e3ee4319a1a5b813e81ccbd9, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679838856016 2023-03-26 13:54:19,743 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55434 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 13:54:19,743 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/B is initiating minor compaction (all files) 2023-03-26 13:54:19,743 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/B in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:19,743 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/06dfd74305b34ec8aa19275ecfe7bae6, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/89ab3dee95ce4b65a14a1d6b6db35edd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/44b1fab7bb964146881fcc5c9c40bd12, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a0027c460d73498f8ff22c201e2c0097, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/96127f7d438a474794cb3d6b65892a7c] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=54.1 K 2023-03-26 13:54:19,743 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,743 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,744 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 33b03d23e9f84ca8990d4f865acab751, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1679838857200 2023-03-26 13:54:19,744 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 06dfd74305b34ec8aa19275ecfe7bae6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679838855389 2023-03-26 13:54:19,744 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 511d71e8beab47e2972c60327aa36174, keycount=50, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1679838858309 2023-03-26 13:54:19,744 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 89ab3dee95ce4b65a14a1d6b6db35edd, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679838856016 2023-03-26 13:54:19,745 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting f63e6a8e0b1148f9936084c736ba6653, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1679838858409 2023-03-26 13:54:19,745 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,745 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:19,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:19,745 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,745 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:19,745 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 44b1fab7bb964146881fcc5c9c40bd12, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1679838857200 2023-03-26 13:54:19,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,746 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting a0027c460d73498f8ff22c201e2c0097, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1679838858309 2023-03-26 13:54:19,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:19,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,746 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 96127f7d438a474794cb3d6b65892a7c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1679838858409 2023-03-26 13:54:19,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:19,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:19,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:19,747 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:19,747 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:19,747 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:19,748 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:19,748 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:19,748 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:19,748 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:19,748 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:19,748 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:19,748 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:19,748 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:19,748 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,748 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:19,748 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:19,750 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,750 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,750 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,750 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:19,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:19,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@89566c0[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:19,753 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:19,753 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,755 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,755 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:19,756 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:19,756 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,756 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:19,756 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:19,757 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,757 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:19,758 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:19,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,758 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:19,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:19,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:19,760 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:19,760 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:19,760 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:19,760 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:19,760 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:19,762 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,762 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,762 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:19,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:19,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:19,763 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,766 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#B#compaction#172 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:19,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,768 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:19,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:19,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:19,771 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:19,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:19,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:19,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:19,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,772 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,772 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:19,772 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:19,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:19,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:19,772 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,773 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,773 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,782 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#A#compaction#171 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:19,872 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:19,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:19,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:19,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:19,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:19,876 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:19,876 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:19,876 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:19,876 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:19,876 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,877 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,879 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,879 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:19,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:19,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:19,879 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,879 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:19,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:19,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:19,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:19,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:19,880 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:19,880 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:19,880 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:19,881 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:19,881 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:19,881 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:19,881 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:19,881 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:19,881 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:19,881 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:19,881 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:19,881 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,881 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:19,881 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:19,883 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,883 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,883 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,883 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:19,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43965235[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:19,885 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:19,886 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,887 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:19,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:19,887 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,887 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:19,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:19,889 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:19,889 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:19,889 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:19,889 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:19,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:19,893 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:19,893 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:19,893 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:19,893 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:19,893 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:19,896 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,896 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,896 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:19,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:19,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:19,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:19,897 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:19,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,898 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:19,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:19,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,903 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,903 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:19,904 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:19,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:19,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:19,904 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:19,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:19,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:19,904 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:19,904 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:19,904 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:19,904 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:19,904 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:19,904 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,904 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:19,905 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:20,012 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:20,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:20,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:20,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:20,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:20,017 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:20,017 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:20,017 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:20,018 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:20,018 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,018 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,019 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,019 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:20,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:20,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:20,020 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,020 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:20,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:20,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,021 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:20,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:20,028 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:20,028 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:20,028 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:20,028 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:20,028 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:20,028 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:20,028 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:20,029 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:20,029 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:20,029 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:20,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:20,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:20,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:20,033 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,033 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,033 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,033 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:20,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:20,035 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:20,035 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8de41c9[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:20,036 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,036 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:20,036 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:20,036 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:20,036 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,036 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:20,037 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,037 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:20,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:20,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:20,041 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:20,041 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:20,041 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:20,041 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:20,041 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:20,043 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,043 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:20,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,043 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:20,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,045 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,046 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,050 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:20,050 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:20,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:20,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:20,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:20,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:20,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:20,050 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,051 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:20,050 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:20,051 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:20,051 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:20,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:20,051 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,051 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,051 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,051 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,151 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:20,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:20,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:20,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:20,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:20,155 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:20,155 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:20,155 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:20,155 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:20,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,157 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:20,157 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:20,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,158 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,158 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:20,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:20,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:20,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:20,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:20,159 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:20,159 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:20,159 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:20,160 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:20,160 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:20,160 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:20,160 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:20,160 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:20,161 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:20,161 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:20,161 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:20,161 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,161 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:20,161 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:20,162 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,162 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,163 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,163 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:20,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64e83b9b[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:20,166 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:20,166 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,167 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,167 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:20,167 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:20,167 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,167 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:20,167 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:20,170 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,170 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:20,170 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,170 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:20,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:20,173 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:20,173 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:20,173 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:20,173 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:20,173 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:20,175 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,175 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:20,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,175 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:20,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,176 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,177 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,180 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:20,180 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:20,180 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,180 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,180 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:20,180 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,180 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:20,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:20,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:20,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:20,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:20,180 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,180 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,180 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,180 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:20,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:20,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:20,180 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:20,195 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/3a835a4081a94491b51b91e627bee0bc as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/3a835a4081a94491b51b91e627bee0bc 2023-03-26 13:54:20,200 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a739eafaf8b68b3135ee013236515a80/B of a739eafaf8b68b3135ee013236515a80 into 3a835a4081a94491b51b91e627bee0bc(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:20,200 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:20,201 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/B, priority=11, startTime=1679838859739; duration=0sec 2023-03-26 13:54:20,201 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:20,201 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 13:54:20,203 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55434 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 13:54:20,203 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/C is initiating minor compaction (all files) 2023-03-26 13:54:20,203 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/C in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:20,203 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/a0ed65be43e446768509dbf67f85237a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/a73bcb7d8c2742788f97bf9b8d310e7f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/c29f7f7247e348aba502696491bc1e67, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/f4cf4bfc80b44a16af435661fcf060b9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/f5d5b77dc9fe4693b7e834e736d83083] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=54.1 K 2023-03-26 13:54:20,203 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting a0ed65be43e446768509dbf67f85237a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679838855389 2023-03-26 13:54:20,204 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting a73bcb7d8c2742788f97bf9b8d310e7f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=134, earliestPutTs=1679838856016 2023-03-26 13:54:20,204 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting c29f7f7247e348aba502696491bc1e67, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1679838857200 2023-03-26 13:54:20,205 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting f4cf4bfc80b44a16af435661fcf060b9, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1679838858309 2023-03-26 13:54:20,205 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting f5d5b77dc9fe4693b7e834e736d83083, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1679838858409 2023-03-26 13:54:20,215 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#C#compaction#173 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:20,281 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:20,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:20,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:20,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:20,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:20,296 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:20,296 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:20,296 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:20,296 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:20,297 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,297 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,298 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,298 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:20,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:20,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:20,298 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,298 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:20,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:20,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:20,301 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:20,301 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:20,301 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:20,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:20,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:20,304 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:20,304 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:20,305 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:20,305 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:20,305 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:20,305 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:20,305 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,305 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:20,305 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:20,307 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,307 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,308 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,308 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:20,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:20,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47e8e2a0[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:20,311 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,311 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:20,311 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:20,311 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:20,311 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,311 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:20,312 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,312 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:20,312 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:20,312 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:20,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:20,315 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:20,315 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:20,315 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:20,315 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:20,315 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:20,317 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,317 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,317 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:20,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:20,319 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,320 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:20,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:20,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:20,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:20,324 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,324 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,324 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,324 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,324 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,324 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:20,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:20,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:20,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:20,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:20,324 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:20,324 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:20,324 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,324 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,424 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:20,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:20,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:20,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:20,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:20,436 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:20,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:20,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:20,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:20,437 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,437 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,441 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:20,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:20,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:20,441 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,442 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,443 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:20,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:20,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:20,444 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:20,444 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:20,444 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:20,456 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:20,456 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:20,456 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:20,456 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:20,456 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:20,456 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:20,456 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:20,456 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:20,456 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,456 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:20,456 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:20,458 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,458 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,459 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,459 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:20,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@361e1792[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:20,463 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:20,463 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,464 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,464 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:20,464 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:20,464 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:20,465 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,465 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:20,467 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:20,467 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:20,467 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:20,467 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:20,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:20,471 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:20,471 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:20,471 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:20,471 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:20,471 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:20,473 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,473 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:20,473 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,475 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,476 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,478 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,478 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:20,478 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,479 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,478 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:20,479 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,479 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,479 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:20,479 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:20,479 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:20,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:20,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:20,479 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:20,479 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:20,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:20,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:20,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:20,577 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:54:20,577 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:20,577 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:20,578 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:20,578 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:20,578 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:20,578 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:20,580 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:20,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:20,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:20,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:20,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:20,585 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:20,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:20,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:20,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:20,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,586 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:20,586 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:20,587 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,587 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:20,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:20,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:20,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:20,592 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:20,592 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:20,592 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:20,594 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:20,594 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:20,594 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:20,594 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:20,594 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:20,595 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:20,595 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:20,595 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:20,595 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,595 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:20,596 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:20,599 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,599 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,599 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,599 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:20,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c34d891[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:20,602 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:20,602 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,603 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,603 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:20,603 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:20,603 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:20,604 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,604 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:20,605 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:20,605 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:20,605 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:20,605 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:20,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:20,611 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:20,611 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:20,611 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:20,611 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:20,611 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:20,613 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,613 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,613 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:20,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:20,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,614 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,616 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679838860578, max timestamp=1679838860616Num uniques 859; 2023-03-26 13:54:20,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679838860578, max timestamp=1679838860616Num uniques 859; 2023-03-26 13:54:20,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-26 13:54:20,618 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-26 13:54:20,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:20,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,621 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:20,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:20,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:20,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:20,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:20,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:20,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,621 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:20,622 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:20,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:20,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:20,623 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303260c541cd74cd34977b17fa6a94994f19a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260c541cd74cd34977b17fa6a94994f19a 2023-03-26 13:54:20,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:20,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838920621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,631 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/b468f7e0253f4c33b5936392568990af as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/b468f7e0253f4c33b5936392568990af 2023-03-26 13:54:20,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:20,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838920624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:20,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838920626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:20,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838920626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:20,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838920637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,638 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a739eafaf8b68b3135ee013236515a80/C of a739eafaf8b68b3135ee013236515a80 into b468f7e0253f4c33b5936392568990af(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:20,638 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:20,638 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/C, priority=11, startTime=1679838859740; duration=0sec 2023-03-26 13:54:20,638 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:20,646 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/9844b4f839324f459d3d4131f82db3fe as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/9844b4f839324f459d3d4131f82db3fe 2023-03-26 13:54:20,650 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a739eafaf8b68b3135ee013236515a80/A of a739eafaf8b68b3135ee013236515a80 into 9844b4f839324f459d3d4131f82db3fe(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:20,650 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:20,650 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/A, priority=11, startTime=1679838859738; duration=0sec 2023-03-26 13:54:20,650 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:20,662 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=231, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/06f6eaa81b5b44f6a80e81753fd0a964 2023-03-26 13:54:20,691 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/7f14380d83334d16aa09af2979b1108a 2023-03-26 13:54:20,722 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:20,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:20,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:20,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:20,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:20,728 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:20,728 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:20,728 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:20,728 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:20,728 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:20,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838920727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,730 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,730 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,731 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,731 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:20,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:20,731 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,731 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:20,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:20,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:20,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:20,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:20,733 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:20,733 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:20,733 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:20,735 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:20,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838920733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:20,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838920734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:20,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838920734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,739 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:20,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838920738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,744 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:20,744 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:20,744 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:20,744 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:20,744 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:20,744 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:20,744 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:20,744 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:20,744 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,744 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:20,744 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:20,746 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,746 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,746 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,746 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:20,748 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:20,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69a5066f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:20,748 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,749 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,749 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,749 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:20,751 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:20,751 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:20,751 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,751 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:20,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:20,756 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:20,756 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:20,756 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:20,756 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:20,756 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:20,757 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,757 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,757 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:20,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,758 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,760 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,764 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:20,764 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:20,764 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,764 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:20,764 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:20,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:20,764 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:20,764 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,764 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,764 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,764 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:20,764 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:20,764 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:20,865 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:20,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:20,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:20,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:20,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:20,868 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:20,868 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:20,868 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:20,868 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:20,868 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,868 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,870 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:20,870 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:20,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:20,870 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,870 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:20,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:20,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,873 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:20,873 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:20,873 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:20,873 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:20,873 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:20,873 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:20,874 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:20,874 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:20,874 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:20,874 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:20,874 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:20,874 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:20,874 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:20,875 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:20,875 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,875 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:20,875 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:20,877 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,877 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,877 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,877 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:20,880 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:20,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69580419[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:20,880 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,884 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,885 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:20,885 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:20,885 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:20,885 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,885 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:20,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,886 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:20,886 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,886 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:20,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:20,889 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:20,889 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:20,889 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:20,890 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:20,890 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:20,891 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,891 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:20,891 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:20,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:20,892 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:20,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,894 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,898 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,898 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,898 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:20,898 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:20,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:20,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:20,898 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:20,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:20,898 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,898 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,899 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:20,899 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:20,898 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:20,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:20,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:20,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:20,899 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:20,899 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:20,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:20,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838920931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:20,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838920936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:20,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838920937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:20,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838920940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:20,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838920940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:20,999 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:21,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:21,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:21,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:21,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:21,003 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:21,003 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:21,003 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:21,003 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:21,003 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,003 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,005 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:21,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:21,005 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:21,006 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,006 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:21,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:21,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:21,007 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:21,007 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:21,007 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:21,007 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:21,007 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:21,007 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:21,007 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:21,007 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:21,007 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:21,007 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:21,007 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:21,007 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,007 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:21,008 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:21,009 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,009 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,009 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,009 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:21,011 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:21,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fbc3c27[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:21,011 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,012 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,012 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:21,012 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:21,012 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:21,014 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,014 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:21,016 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,016 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:21,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,016 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:21,016 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:21,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:21,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:21,019 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:21,019 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:21,019 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:21,019 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:21,019 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:21,020 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,020 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:21,020 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:21,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:21,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,021 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,022 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,024 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,024 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,024 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,024 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,025 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:21,025 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:21,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:21,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:21,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:21,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:21,025 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,025 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:21,025 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:21,025 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:21,025 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,025 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:21,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:21,115 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/e12d3c8da70a450687d0089e63df6363 2023-03-26 13:54:21,120 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/06f6eaa81b5b44f6a80e81753fd0a964 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/06f6eaa81b5b44f6a80e81753fd0a964 2023-03-26 13:54:21,124 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/06f6eaa81b5b44f6a80e81753fd0a964, entries=350, sequenceid=231, filesize=52.6 K 2023-03-26 13:54:21,125 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/7f14380d83334d16aa09af2979b1108a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/7f14380d83334d16aa09af2979b1108a 2023-03-26 13:54:21,125 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:21,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:21,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:21,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:21,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:21,130 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:21,130 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:21,130 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:21,130 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:21,130 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,130 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,131 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,131 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:21,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:21,132 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,132 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,132 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/7f14380d83334d16aa09af2979b1108a, entries=150, sequenceid=231, filesize=11.8 K 2023-03-26 13:54:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:21,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:21,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:21,133 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:21,133 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:21,133 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:21,133 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/e12d3c8da70a450687d0089e63df6363 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/e12d3c8da70a450687d0089e63df6363 2023-03-26 13:54:21,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:21,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:21,136 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:21,136 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:21,136 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:21,136 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:21,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:21,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:21,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:21,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:21,138 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/e12d3c8da70a450687d0089e63df6363, entries=150, sequenceid=231, filesize=11.8 K 2023-03-26 13:54:21,138 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=154.31 KB/158010 for a739eafaf8b68b3135ee013236515a80 in 561ms, sequenceid=231, compaction requested=false 2023-03-26 13:54:21,138 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:21,140 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,140 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,142 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,142 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:21,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:21,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45a68911[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:21,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:21,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,143 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,144 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:21,144 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:21,144 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,144 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:21,144 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:21,145 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,145 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:21,145 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,145 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:21,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:21,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:21,148 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:21,148 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:21,148 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:21,148 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:21,148 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:21,150 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,151 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,152 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,157 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,157 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:21,157 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,157 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:21,157 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,157 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,157 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,157 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:21,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:21,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:21,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,158 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:21,158 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:21,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:21,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:21,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:21,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:21,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:21,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:21,240 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=161.02 KB heapSize=425.89 KB 2023-03-26 13:54:21,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:21,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:21,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:21,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:21,242 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:21,242 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:21,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838921243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,246 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838921244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838921246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838921246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838921248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,257 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326e39abc346fd64fdb9adc39bc8e4997d7 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e39abc346fd64fdb9adc39bc8e4997d7 2023-03-26 13:54:21,258 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:21,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:21,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:21,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:21,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:21,261 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:21,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:21,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:21,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:21,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,263 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:21,263 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:21,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:21,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:21,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:21,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:21,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:21,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:21,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:21,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:21,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:21,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:21,265 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:21,266 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:21,266 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=262, memsize=58.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/b67efe2cff6244faa1780a08dee1517c 2023-03-26 13:54:21,266 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:21,266 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:21,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:21,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:21,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:21,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:21,268 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:21,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:21,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f169b4[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:21,271 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:21,271 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,273 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,273 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:21,281 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,281 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:21,281 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:21,281 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:21,285 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,285 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:21,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,286 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:21,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:21,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:21,289 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:21,289 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:21,289 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:21,289 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:21,289 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:21,290 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,290 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,290 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:21,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,291 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,292 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:21,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,295 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:21,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:21,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:21,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:21,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:21,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:21,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:21,295 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:21,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:21,296 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:21,299 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/a38f4a123b424eae9f58a37eed4c109c 2023-03-26 13:54:21,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838921347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838921350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838921350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,396 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:21,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:21,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:21,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:21,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:21,399 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:21,399 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:21,399 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:21,399 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:21,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,402 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,402 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:21,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:21,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:21,402 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,403 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:21,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:21,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:21,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:21,404 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:21,404 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:21,404 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:21,404 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:21,405 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:21,405 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:21,405 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,405 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:21,405 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:21,406 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,406 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,406 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,406 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:21,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2da0cdff[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:21,408 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:21,408 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,409 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:21,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:21,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:21,409 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,409 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:21,411 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,411 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:21,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:21,413 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:21,413 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:21,413 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:21,413 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:21,413 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:21,415 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,415 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:21,415 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,416 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,417 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,419 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:21,419 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:21,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:21,419 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:21,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:21,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:21,420 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:21,420 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,420 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,420 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,420 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:21,420 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:21,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:21,420 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:21,420 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:21,420 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,520 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:21,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:21,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:21,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:21,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:21,523 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:21,523 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:21,523 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:21,523 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:21,524 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,524 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,525 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:21,525 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,526 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,526 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:21,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,527 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:21,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:21,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:21,528 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:21,528 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:21,528 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:21,528 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:21,528 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:21,528 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:21,528 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:21,528 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:21,528 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:21,530 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,530 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,530 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,531 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:21,533 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:21,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63b38168[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:21,533 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,535 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:21,535 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:21,535 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:21,535 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,535 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:21,536 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,536 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:21,536 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,536 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:21,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:21,539 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:21,539 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:21,539 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:21,539 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:21,539 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:21,540 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,540 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,540 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,541 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,543 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,545 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:21,545 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:21,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:21,545 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,546 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,546 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,546 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:21,546 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:21,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:21,546 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,546 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,546 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:21,546 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:21,546 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:21,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:21,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838921550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838921553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838921554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,646 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:21,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:21,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:21,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:21,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:21,650 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:21,650 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:21,650 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:21,650 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:21,651 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,651 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,653 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:21,653 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:21,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:21,654 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,654 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:21,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:21,655 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:21,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:21,655 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:21,655 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:21,655 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:21,655 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,655 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:21,656 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:21,657 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,657 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,657 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,658 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:21,659 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:21,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@122c720e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:21,659 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,660 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,660 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:21,660 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:21,660 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:21,661 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,661 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:21,662 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:21,662 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:21,662 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,662 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:21,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:21,666 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:21,666 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:21,666 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:21,666 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:21,666 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:21,668 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,668 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,668 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:21,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:21,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:21,669 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,670 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,675 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,675 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,675 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,675 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:21,675 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:21,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:21,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:21,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:21,675 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:21,675 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,676 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,676 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,676 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,676 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:21,676 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:21,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:21,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:21,712 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=262 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/8811323f7fdc468081f15e60f6d34d5d 2023-03-26 13:54:21,716 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/b67efe2cff6244faa1780a08dee1517c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/b67efe2cff6244faa1780a08dee1517c 2023-03-26 13:54:21,720 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/b67efe2cff6244faa1780a08dee1517c, entries=200, sequenceid=262, filesize=32.3 K 2023-03-26 13:54:21,721 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/a38f4a123b424eae9f58a37eed4c109c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a38f4a123b424eae9f58a37eed4c109c 2023-03-26 13:54:21,725 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a38f4a123b424eae9f58a37eed4c109c, entries=150, sequenceid=262, filesize=11.9 K 2023-03-26 13:54:21,726 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/8811323f7fdc468081f15e60f6d34d5d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/8811323f7fdc468081f15e60f6d34d5d 2023-03-26 13:54:21,737 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/8811323f7fdc468081f15e60f6d34d5d, entries=150, sequenceid=262, filesize=11.9 K 2023-03-26 13:54:21,738 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~174.43 KB/178620, heapSize ~465.69 KB/476864, currentSize=20.13 KB/20610 for a739eafaf8b68b3135ee013236515a80 in 498ms, sequenceid=262, compaction requested=true 2023-03-26 13:54:21,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:21,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:21,738 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:21,738 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:21,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:21,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:21,741 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36618 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:21,741 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 113202 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:21,741 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/B is initiating minor compaction (all files) 2023-03-26 13:54:21,741 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/A is initiating minor compaction (all files) 2023-03-26 13:54:21,741 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/B in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:21,741 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/3a835a4081a94491b51b91e627bee0bc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/7f14380d83334d16aa09af2979b1108a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a38f4a123b424eae9f58a37eed4c109c] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=35.8 K 2023-03-26 13:54:21,741 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/A in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:21,742 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/9844b4f839324f459d3d4131f82db3fe, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/06f6eaa81b5b44f6a80e81753fd0a964, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/b67efe2cff6244faa1780a08dee1517c] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=110.5 K 2023-03-26 13:54:21,742 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 3a835a4081a94491b51b91e627bee0bc, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1679838858409 2023-03-26 13:54:21,742 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 9844b4f839324f459d3d4131f82db3fe, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1679838858409 2023-03-26 13:54:21,742 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 7f14380d83334d16aa09af2979b1108a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1679838860567 2023-03-26 13:54:21,743 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 06f6eaa81b5b44f6a80e81753fd0a964, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1679838860534 2023-03-26 13:54:21,745 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting a38f4a123b424eae9f58a37eed4c109c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1679838860624 2023-03-26 13:54:21,745 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting b67efe2cff6244faa1780a08dee1517c, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1679838860624 2023-03-26 13:54:21,756 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#B#compaction#181 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:21,770 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#A#compaction#180 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:21,776 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:21,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:21,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:21,782 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:54:21,782 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:21,782 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:21,782 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:21,782 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:21,782 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:21,782 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:21,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:21,786 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/78a693a12d7b466a96a7ddb162878939 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/78a693a12d7b466a96a7ddb162878939 2023-03-26 13:54:21,796 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a739eafaf8b68b3135ee013236515a80/B of a739eafaf8b68b3135ee013236515a80 into 78a693a12d7b466a96a7ddb162878939(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:21,796 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:21,796 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/B, priority=13, startTime=1679838861738; duration=0sec 2023-03-26 13:54:21,796 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:21,796 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:21,797 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36618 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:21,798 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/C is initiating minor compaction (all files) 2023-03-26 13:54:21,798 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/C in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:21,798 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/b468f7e0253f4c33b5936392568990af, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/e12d3c8da70a450687d0089e63df6363, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/8811323f7fdc468081f15e60f6d34d5d] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=35.8 K 2023-03-26 13:54:21,798 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting b468f7e0253f4c33b5936392568990af, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1679838858409 2023-03-26 13:54:21,799 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e12d3c8da70a450687d0089e63df6363, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1679838860567 2023-03-26 13:54:21,799 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 8811323f7fdc468081f15e60f6d34d5d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1679838860624 2023-03-26 13:54:21,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:21,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:21,800 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:21,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:21,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:21,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:21,801 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,801 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,802 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:21,802 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,803 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,803 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:21,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:21,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:21,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:21,812 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:21,812 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:21,812 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:21,812 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:21,812 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:21,812 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:21,812 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:21,812 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:21,812 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,829 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#C#compaction#183 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:21,836 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:21,836 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:21,838 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,838 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,838 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,838 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:21,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d9f1536[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:21,840 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:21,840 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,843 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:21,843 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:21,843 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,843 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:21,844 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:21,844 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:21,844 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:21,844 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:21,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:21,848 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:21,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:21,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:21,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:21,848 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:21,850 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,850 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:21,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,850 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:21,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:21,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:21,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,852 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:21,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:21,856 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838861841, max timestamp=1679838861855Num uniques 859; 2023-03-26 13:54:21,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:54:21,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838861841, max timestamp=1679838861855Num uniques 859; 2023-03-26 13:54:21,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:54:21,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838861841, max timestamp=1679838861855Num uniques 859; 2023-03-26 13:54:21,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:54:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:21,863 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:21,863 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:21,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:21,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:21,864 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,864 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,864 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,864 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,864 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,864 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:21,864 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,864 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:21,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:21,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:21,865 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:21,865 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:21,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:21,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:21,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:21,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838921875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40054 deadline: 1679838921876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838921878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40040 deadline: 1679838921882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40020 deadline: 1679838921882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,891 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032637f853ebb44c48a58a8750ff8fbae3f6 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032637f853ebb44c48a58a8750ff8fbae3f6 2023-03-26 13:54:21,900 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/31073bd57dc549eb9ed4548bf11584cb as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/31073bd57dc549eb9ed4548bf11584cb 2023-03-26 13:54:21,904 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/ba410f580e7d495e8b38d565b6703465 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/ba410f580e7d495e8b38d565b6703465 2023-03-26 13:54:21,909 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a739eafaf8b68b3135ee013236515a80/C of a739eafaf8b68b3135ee013236515a80 into 31073bd57dc549eb9ed4548bf11584cb(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:21,909 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:21,909 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/C, priority=13, startTime=1679838861738; duration=0sec 2023-03-26 13:54:21,909 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:21,911 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a739eafaf8b68b3135ee013236515a80/A of a739eafaf8b68b3135ee013236515a80 into ba410f580e7d495e8b38d565b6703465(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:21,911 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:21,911 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/A, priority=13, startTime=1679838861738; duration=0sec 2023-03-26 13:54:21,911 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:21,918 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=272, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/2720d727e11f4be09854792776fe8d03 2023-03-26 13:54:21,935 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/779bf18501894f5985d67ace90fe8bb0 2023-03-26 13:54:21,985 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:21,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:21,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:21,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:21,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:21,990 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:21,990 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:21,990 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:21,990 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:21,990 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,990 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838921986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838921986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838921987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,992 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838921988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:21,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838921988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,993 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:21,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:21,993 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,993 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:21,993 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:21,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:21,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:22,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:22,000 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:22,000 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:22,000 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:22,000 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:22,000 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:22,001 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:22,001 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:22,001 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:22,001 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:22,001 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:22,001 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:22,001 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,001 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:22,001 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:22,003 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,003 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,005 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,005 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:22,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:22,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36fbfc4f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:22,006 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:22,006 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,007 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,007 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:22,008 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:22,008 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:22,008 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,008 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:22,010 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,011 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:22,011 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:22,011 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:22,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:22,013 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:22,013 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:22,013 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:22,013 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:22,013 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:22,016 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,016 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,016 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:22,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,018 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,020 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,022 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,022 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:22,022 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:22,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:22,022 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,022 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:22,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:22,022 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:22,023 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:22,023 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,023 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:22,023 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,023 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,023 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:22,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:22,123 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:22,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:22,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:22,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:22,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:22,126 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:22,126 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:22,126 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:22,126 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:22,126 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,127 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,128 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:22,128 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:22,128 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,128 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:22,129 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:22,129 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:22,129 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:22,129 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:22,129 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:22,129 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:22,129 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:22,129 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:22,130 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:22,130 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:22,130 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:22,130 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:22,130 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:22,130 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,130 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:22,130 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:22,131 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,131 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,131 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,131 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:22,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:22,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@493c8339[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:22,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:22,135 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:22,135 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,135 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:22,136 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:22,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:22,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:22,139 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:22,139 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:22,139 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:22,139 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:22,139 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:22,140 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,140 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:22,140 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:22,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,144 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:22,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:22,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:22,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:22,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:22,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:22,152 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:22,152 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:22,152 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:22,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:22,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:22,193 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838922193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838922193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40010 deadline: 1679838922193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838922194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838922194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,252 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:22,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:22,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:22,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:22,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:22,256 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:22,256 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:22,256 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:22,256 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:22,256 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,257 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,258 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,258 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:22,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:22,258 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,258 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:22,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:22,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:22,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:22,260 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:22,260 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:22,260 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:22,260 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:22,260 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:22,260 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:22,260 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:22,260 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:22,260 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:22,260 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:22,260 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:22,260 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,260 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:22,260 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:22,262 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,262 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,262 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,262 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:22,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:22,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38713412[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:22,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,265 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:22,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:22,266 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,266 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:22,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:22,268 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:22,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:22,270 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:22,270 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:22,270 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:22,270 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:22,270 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:22,272 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,272 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,272 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:22,272 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,274 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:22,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:22,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:22,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:22,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:22,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:22,276 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:22,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:22,276 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:22,277 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:22,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:22,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:22,377 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:22,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:22,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:22,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:22,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:22,380 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:22,380 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:22,380 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:22,380 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:22,380 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,381 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,382 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:22,382 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:22,382 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,382 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:22,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:22,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:22,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:22,384 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:22,384 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:22,384 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:22,388 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:22,388 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:22,388 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:22,388 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:22,388 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:22,388 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:22,388 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:22,388 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:22,388 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,388 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:22,388 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:22,389 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,390 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,390 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,390 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:22,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:22,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d22b08f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:22,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,392 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,392 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:22,393 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:22,393 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:22,393 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,393 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:22,395 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,395 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:22,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,395 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:22,395 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:22,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:22,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:22,397 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:22,397 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:22,397 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:22,397 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:22,397 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:22,398 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,398 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,398 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:22,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:22,399 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,401 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=272 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/b847397582af4a1db256af03466e7989 2023-03-26 13:54:22,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,403 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:22,403 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:22,403 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,403 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,403 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,403 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,403 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,403 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,403 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:22,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:22,403 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:22,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:22,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:22,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:22,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:22,404 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:22,404 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:22,407 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/2720d727e11f4be09854792776fe8d03 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/2720d727e11f4be09854792776fe8d03 2023-03-26 13:54:22,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:22,412 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/2720d727e11f4be09854792776fe8d03, entries=200, sequenceid=272, filesize=32.3 K 2023-03-26 13:54:22,413 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/779bf18501894f5985d67ace90fe8bb0 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/779bf18501894f5985d67ace90fe8bb0 2023-03-26 13:54:22,417 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/779bf18501894f5985d67ace90fe8bb0, entries=100, sequenceid=272, filesize=9.5 K 2023-03-26 13:54:22,418 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/b847397582af4a1db256af03466e7989 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/b847397582af4a1db256af03466e7989 2023-03-26 13:54:22,422 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/b847397582af4a1db256af03466e7989, entries=100, sequenceid=272, filesize=9.5 K 2023-03-26 13:54:22,423 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for a739eafaf8b68b3135ee013236515a80 in 642ms, sequenceid=272, compaction requested=false 2023-03-26 13:54:22,423 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:22,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:22,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679838861865, max timestamp=1679838862496Num uniques 1068; 2023-03-26 13:54:22,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-26 13:54:22,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679838861865, max timestamp=1679838862496Num uniques 1068; 2023-03-26 13:54:22,499 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-26 13:54:22,500 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679838861865, max timestamp=1679838862496Num uniques 1068; 2023-03-26 13:54:22,500 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-26 13:54:22,500 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=46.96 KB heapSize=125.16 KB 2023-03-26 13:54:22,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:22,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:22,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:22,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:22,501 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:22,501 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:22,504 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:22,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:22,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:22,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:22,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:22,512 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:22,512 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:22,512 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:22,512 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:22,512 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,513 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,514 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:22,514 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,514 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,514 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:22,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:22,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:22,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:22,515 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:22,515 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:22,515 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:22,515 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:22,516 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:22,516 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:22,516 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:22,516 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:22,516 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:22,516 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:22,516 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:22,516 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,516 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:22,516 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:22,523 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,523 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,523 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,523 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:22,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838862517, max timestamp=1679838862523Num uniques 859; 2023-03-26 13:54:22,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:54:22,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838862517, max timestamp=1679838862523Num uniques 859; 2023-03-26 13:54:22,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838862517, max timestamp=1679838862523Num uniques 859; 2023-03-26 13:54:22,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:54:22,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:54:22,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:22,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34d236e0[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:22,526 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:22,526 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,527 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:22,527 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,527 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:22,528 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,528 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:22,528 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:22,529 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,535 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032681302e36bec44698acf7dfbc6ad88458 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032681302e36bec44698acf7dfbc6ad88458 2023-03-26 13:54:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:22,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:22,545 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:22,545 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:22,545 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:22,545 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:22,545 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:22,547 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,547 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:22,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,547 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,548 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,550 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,553 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:22,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:22,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:22,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:22,554 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:22,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:22,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:22,554 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:22,554 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:22,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:22,562 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=317, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/145f162063df4e89bca44bb6c741e7f5 2023-03-26 13:54:22,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838922552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838922568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838922568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838922568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838922568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,580 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/2dc71f187f374347b17400753ae2a8f9 2023-03-26 13:54:22,654 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:22,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:22,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:22,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:22,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:22,658 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:22,658 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:22,658 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:22,658 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:22,659 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,659 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,663 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:22,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:22,663 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,664 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,664 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:22,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:22,665 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:22,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:22,666 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:22,666 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:22,666 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:22,666 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:22,666 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:22,668 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,668 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:22,671 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:22,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@195a5e08[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:22,671 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838922669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,674 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,674 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:22,674 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:22,674 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:22,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838922674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838922674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838922674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838922675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,676 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,676 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:22,677 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:22,677 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:22,677 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,677 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:22,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:22,682 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:22,682 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:22,682 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:22,682 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:22,682 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:22,684 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,687 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,687 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:22,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,687 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:22,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:22,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,691 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:22,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:22,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:22,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:22,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:22,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:22,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,692 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:22,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:22,692 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:22,792 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:22,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:22,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:22,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:22,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:22,800 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:22,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:22,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:22,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:22,801 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,801 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,803 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:22,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:22,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:22,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:22,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:22,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:22,805 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:22,805 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:22,805 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:22,806 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,807 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,807 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:22,816 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:22,816 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:22,816 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:22,816 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:22,817 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:22,817 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:22,817 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:22,817 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:22,817 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,817 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:22,817 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:22,819 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,819 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,819 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,819 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:22,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28ba3895[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:22,823 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:22,823 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,824 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:22,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:22,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:22,824 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,824 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:22,826 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:22,826 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:22,826 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:22,827 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:22,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:22,830 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:22,830 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:22,830 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:22,830 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:22,830 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:22,832 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,832 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:22,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,832 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:22,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:22,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,833 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,838 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:22,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:22,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:22,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:22,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:22,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,842 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:22,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:22,843 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:22,843 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:22,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:22,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:22,877 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838922875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838922876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838922877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838922877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:22,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838922878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,943 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:22,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:22,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:22,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:22,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:22,947 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:22,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:22,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:22,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:22,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,948 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,949 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:22,949 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:22,949 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,949 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:22,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,950 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:22,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:22,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:22,951 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:22,951 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:22,951 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:22,956 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:22,956 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:22,956 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:22,956 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:22,956 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:22,956 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:22,956 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:22,956 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:22,956 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,957 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:22,957 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:22,958 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:22,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60d443c5[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:22,960 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:22,960 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,961 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,961 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:22,962 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,962 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:22,962 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,962 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:22,962 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:22,963 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:22,964 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:22,964 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:22,964 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:22,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:22,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:22,971 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:22,971 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:22,971 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:22,971 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:22,971 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:22,973 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,973 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:22,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,973 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:22,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,976 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:22,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:22,977 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:22,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,980 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,980 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:22,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:22,980 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:22,980 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:22,980 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:22,980 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:22,980 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:22,980 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:22,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:22,980 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:22,980 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,980 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:22,980 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:22,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:22,992 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=317 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/56a5332896ed4e70ab6c0f53f18ed9f8 2023-03-26 13:54:23,002 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/145f162063df4e89bca44bb6c741e7f5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/145f162063df4e89bca44bb6c741e7f5 2023-03-26 13:54:23,010 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/145f162063df4e89bca44bb6c741e7f5, entries=200, sequenceid=317, filesize=32.3 K 2023-03-26 13:54:23,011 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/2dc71f187f374347b17400753ae2a8f9 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/2dc71f187f374347b17400753ae2a8f9 2023-03-26 13:54:23,015 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/2dc71f187f374347b17400753ae2a8f9, entries=150, sequenceid=317, filesize=11.9 K 2023-03-26 13:54:23,017 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/56a5332896ed4e70ab6c0f53f18ed9f8 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/56a5332896ed4e70ab6c0f53f18ed9f8 2023-03-26 13:54:23,022 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/56a5332896ed4e70ab6c0f53f18ed9f8, entries=150, sequenceid=317, filesize=11.9 K 2023-03-26 13:54:23,023 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.03 KB/148512, currentSize=147.60 KB/151140 for a739eafaf8b68b3135ee013236515a80 in 523ms, sequenceid=317, compaction requested=true 2023-03-26 13:54:23,023 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:23,023 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:23,024 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:23,024 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:23,024 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:23,024 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:23,025 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92676 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:23,025 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/A is initiating minor compaction (all files) 2023-03-26 13:54:23,025 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/A in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:23,025 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/ba410f580e7d495e8b38d565b6703465, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/2720d727e11f4be09854792776fe8d03, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/145f162063df4e89bca44bb6c741e7f5] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=90.5 K 2023-03-26 13:54:23,026 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting ba410f580e7d495e8b38d565b6703465, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1679838860624 2023-03-26 13:54:23,026 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34576 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:23,026 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/B is initiating minor compaction (all files) 2023-03-26 13:54:23,026 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/B in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:23,026 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/78a693a12d7b466a96a7ddb162878939, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/779bf18501894f5985d67ace90fe8bb0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/2dc71f187f374347b17400753ae2a8f9] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=33.8 K 2023-03-26 13:54:23,026 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 2720d727e11f4be09854792776fe8d03, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1679838861771 2023-03-26 13:54:23,027 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 78a693a12d7b466a96a7ddb162878939, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1679838860624 2023-03-26 13:54:23,027 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 145f162063df4e89bca44bb6c741e7f5, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679838861880 2023-03-26 13:54:23,027 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 779bf18501894f5985d67ace90fe8bb0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1679838861780 2023-03-26 13:54:23,028 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 2dc71f187f374347b17400753ae2a8f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679838861880 2023-03-26 13:54:23,044 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#B#compaction#189 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:23,061 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#A#compaction#190 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:23,080 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:23,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:23,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:23,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:23,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:23,088 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:23,088 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:23,088 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:23,088 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:23,088 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:23,088 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:23,091 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:23,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:23,091 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:23,091 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,091 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:23,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:23,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:23,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:23,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:23,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:23,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:23,096 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:23,096 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:23,096 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:23,104 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:23,104 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:23,104 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:23,104 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:23,105 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-03-26 13:54:23,105 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:23,105 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:23,105 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:23,105 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:23,105 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:23,105 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:23,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:23,184 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:23,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838923185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838923187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838923188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838923188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838923189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838923290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838923290, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838923293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,296 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838923294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838923294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,469 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/386d137d738542618e5ebfed757401fd as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/386d137d738542618e5ebfed757401fd 2023-03-26 13:54:23,474 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a739eafaf8b68b3135ee013236515a80/B of a739eafaf8b68b3135ee013236515a80 into 386d137d738542618e5ebfed757401fd(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:23,474 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:23,474 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/B, priority=13, startTime=1679838863024; duration=0sec 2023-03-26 13:54:23,474 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:23,474 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:23,475 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34576 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:23,475 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/C is initiating minor compaction (all files) 2023-03-26 13:54:23,476 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/C in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:23,476 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/31073bd57dc549eb9ed4548bf11584cb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/b847397582af4a1db256af03466e7989, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/56a5332896ed4e70ab6c0f53f18ed9f8] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=33.8 K 2023-03-26 13:54:23,477 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 31073bd57dc549eb9ed4548bf11584cb, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=262, earliestPutTs=1679838860624 2023-03-26 13:54:23,477 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting b847397582af4a1db256af03466e7989, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=272, earliestPutTs=1679838861780 2023-03-26 13:54:23,477 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 56a5332896ed4e70ab6c0f53f18ed9f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679838861880 2023-03-26 13:54:23,485 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#C#compaction#192 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:23,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838923494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838923494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838923497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838923497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838923498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,550 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032656f1d3a697384a2e95df940b4232c573 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032656f1d3a697384a2e95df940b4232c573 2023-03-26 13:54:23,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838923798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838923799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838923800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838923800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,805 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:23,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838923801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:23,896 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/2de3ca8d707e488a95b99b2d0a0a58d4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/2de3ca8d707e488a95b99b2d0a0a58d4 2023-03-26 13:54:23,906 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/7796cf7ae41244e58cb3781708073c42 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/7796cf7ae41244e58cb3781708073c42 2023-03-26 13:54:23,912 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a739eafaf8b68b3135ee013236515a80/A of a739eafaf8b68b3135ee013236515a80 into 7796cf7ae41244e58cb3781708073c42(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:23,912 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:23,912 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/A, priority=13, startTime=1679838863023; duration=0sec 2023-03-26 13:54:23,912 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:23,918 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a739eafaf8b68b3135ee013236515a80/C of a739eafaf8b68b3135ee013236515a80 into 2de3ca8d707e488a95b99b2d0a0a58d4(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:23,918 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:23,918 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/C, priority=13, startTime=1679838863024; duration=0sec 2023-03-26 13:54:23,918 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:23,957 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=354, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/a71a5ccb08f04594bb58e655e8dea68c 2023-03-26 13:54:23,983 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/193baf37023d459bb6bf637683b6469c 2023-03-26 13:54:24,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:24,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838924302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:24,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838924306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:24,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838924307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:24,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838924307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:24,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838924310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,396 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=354 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/b5203f851f80423ca53e3ce37bb7465c 2023-03-26 13:54:24,401 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/a71a5ccb08f04594bb58e655e8dea68c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/a71a5ccb08f04594bb58e655e8dea68c 2023-03-26 13:54:24,405 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/a71a5ccb08f04594bb58e655e8dea68c, entries=150, sequenceid=354, filesize=25.4 K 2023-03-26 13:54:24,406 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/193baf37023d459bb6bf637683b6469c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/193baf37023d459bb6bf637683b6469c 2023-03-26 13:54:24,410 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/193baf37023d459bb6bf637683b6469c, entries=150, sequenceid=354, filesize=11.9 K 2023-03-26 13:54:24,411 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/b5203f851f80423ca53e3ce37bb7465c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/b5203f851f80423ca53e3ce37bb7465c 2023-03-26 13:54:24,415 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/b5203f851f80423ca53e3ce37bb7465c, entries=150, sequenceid=354, filesize=11.9 K 2023-03-26 13:54:24,416 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=40.25 KB/41220 for a739eafaf8b68b3135ee013236515a80 in 1311ms, sequenceid=354, compaction requested=false 2023-03-26 13:54:24,416 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:24,416 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:24,416 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:24,416 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:24,416 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,416 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:24,416 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:24,418 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:24,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:24,419 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:24,419 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:24,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:24,419 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:24,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45141c09[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:24,420 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:24,420 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:24,425 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:24,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:24,425 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:24,425 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:24,425 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:24,425 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,425 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:24,427 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,427 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:24,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:24,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:24,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:24,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:24,430 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:24,430 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:24,430 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:24,430 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:24,430 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:24,432 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:24,432 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:24,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:24,432 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:24,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:24,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:24,433 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:24,434 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,436 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:24,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:24,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,437 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:24,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:24,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:24,437 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:24,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:24,437 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:24,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:24,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:24,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:24,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:24,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:24,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:24,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:24,537 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:24,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:24,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:24,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:24,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:24,540 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:24,540 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:24,540 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:24,540 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:24,541 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:24,541 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:24,542 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:24,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:24,542 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:24,542 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:24,543 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:24,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:24,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:24,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:24,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:24,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:24,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:24,543 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:24,543 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:24,544 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:24,544 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:24,544 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:24,544 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-26 13:54:24,544 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:24,544 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:24,544 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:24,544 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:24,544 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:24,544 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:24,562 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303268daf3607f1c44b8caff7b21150665c8f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303268daf3607f1c44b8caff7b21150665c8f 2023-03-26 13:54:24,968 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=366, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/fee5081a5dd6417a89378994cbbb62d3 2023-03-26 13:54:24,985 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/c1cec8fed9ba49f784d264d5b2db7690 2023-03-26 13:54:25,021 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=366 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/e636bcbb84f84556b105ca1f97e883cc 2023-03-26 13:54:25,026 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/fee5081a5dd6417a89378994cbbb62d3 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/fee5081a5dd6417a89378994cbbb62d3 2023-03-26 13:54:25,031 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/fee5081a5dd6417a89378994cbbb62d3, entries=150, sequenceid=366, filesize=25.4 K 2023-03-26 13:54:25,033 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/c1cec8fed9ba49f784d264d5b2db7690 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/c1cec8fed9ba49f784d264d5b2db7690 2023-03-26 13:54:25,041 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/c1cec8fed9ba49f784d264d5b2db7690, entries=150, sequenceid=366, filesize=11.9 K 2023-03-26 13:54:25,042 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/e636bcbb84f84556b105ca1f97e883cc as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/e636bcbb84f84556b105ca1f97e883cc 2023-03-26 13:54:25,047 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/e636bcbb84f84556b105ca1f97e883cc, entries=150, sequenceid=366, filesize=11.9 K 2023-03-26 13:54:25,049 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for a739eafaf8b68b3135ee013236515a80 in 505ms, sequenceid=366, compaction requested=true 2023-03-26 13:54:25,049 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:25,049 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:25,049 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:25,049 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:25,049 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,049 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:25,049 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:25,052 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,052 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,053 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,053 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:25,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ef5b0bd[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:25,055 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:25,055 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,056 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,056 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:25,056 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:25,056 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:25,057 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,057 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:25,058 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:25,058 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:25,058 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:25,058 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:25,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:25,106 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:25,106 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:25,106 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:25,106 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:25,106 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:25,109 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,109 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:25,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,109 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:25,110 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,112 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,116 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:25,116 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:25,116 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:25,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:25,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:25,116 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:25,117 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:25,117 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:25,117 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:25,117 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,117 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,117 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,117 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,117 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:25,217 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:25,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:25,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:25,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:25,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:25,224 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:25,224 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:25,224 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:25,224 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:25,224 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,225 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,226 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:25,226 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:25,226 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,226 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:25,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:25,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:25,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:25,228 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:25,228 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:25,228 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:25,228 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:25,228 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:25,228 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:25,228 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:25,228 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:25,228 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:25,229 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:25,230 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,230 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,230 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,231 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:25,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77f24f66[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:25,233 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:25,233 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,234 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,234 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:25,235 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:25,235 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:25,235 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,235 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:25,237 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,245 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:25,245 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:25,245 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:25,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:25,247 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:25,247 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:25,247 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:25,247 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:25,247 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:25,249 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,249 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,249 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:25,250 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,251 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,259 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,259 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:25,259 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:25,259 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,259 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:25,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:25,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:25,259 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,260 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:25,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:25,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:25,260 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:25,260 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,260 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:25,260 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:25,260 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:25,350 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:54:25,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:25,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:25,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:25,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:25,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:25,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:25,360 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:25,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:25,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:25,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:25,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:25,376 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:25,376 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:25,376 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:25,376 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:25,377 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,377 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,378 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:25,378 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,379 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,379 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:25,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:25,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,379 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:25,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:25,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:25,381 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:25,381 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:25,381 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:25,388 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:25,388 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:25,389 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:25,389 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:25,389 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:25,389 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:25,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39334, totalHeapSize=89576, min timestamp=1679838865351, max timestamp=1679838865388Num uniques 859; 2023-03-26 13:54:25,391 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=39334, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-26 13:54:25,391 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679838865351, max timestamp=1679838865388Num uniques 859; 2023-03-26 13:54:25,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679838865351, max timestamp=1679838865388Num uniques 859; 2023-03-26 13:54:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,392 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-26 13:54:25,392 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-26 13:54:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,392 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,393 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:25,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35faf036[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:25,394 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:25,394 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,395 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,395 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,396 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:25,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:25,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:25,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:25,400 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,400 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:25,400 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:25,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,400 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:25,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:25,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:25,407 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:25,407 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:25,407 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:25,407 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:25,407 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:25,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:25,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838925404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:25,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838925405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:25,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838925406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,413 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,413 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:25,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,413 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838925407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:25,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838925407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,414 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,416 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,418 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:25,418 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,418 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:25,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:25,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:25,418 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:25,419 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:25,419 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:25,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:25,418 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:25,418 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,419 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,419 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,419 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,419 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:25,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:25,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:25,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:25,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838925509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:25,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838925514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:25,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838925514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:25,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838925515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,518 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:25,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838925515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,519 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:25,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:25,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:25,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:25,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:25,525 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:25,525 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:25,525 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:25,525 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:25,525 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,525 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,527 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:25,527 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:25,528 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,528 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:25,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:25,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:25,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:25,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:25,532 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:25,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:25,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:25,536 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:25,536 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:25,536 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:25,536 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:25,536 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:25,536 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:25,536 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:25,536 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:25,537 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,537 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:25,537 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:25,538 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,538 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,539 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,539 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:25,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:25,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@506b56d4[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:25,542 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:25,542 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,544 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,545 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:25,545 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:25,545 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:25,547 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:25,547 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,547 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:25,547 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:25,547 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:25,547 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:25,547 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:25,548 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:25,548 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:25,548 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:25,548 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:25,549 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,549 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:25,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:25,550 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,550 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,551 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,554 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,554 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,554 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,554 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:25,554 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:25,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:25,554 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,554 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,554 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,555 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:25,555 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:25,555 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:25,555 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:25,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:25,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:25,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:25,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:25,655 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:25,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:25,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:25,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:25,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:25,659 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:25,659 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:25,659 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:25,659 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:25,661 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,661 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,667 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:25,667 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:25,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:25,667 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,668 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:25,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:25,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:25,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:25,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:25,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:25,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:25,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:25,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:25,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:25,669 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:25,669 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:25,669 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:25,669 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:25,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:25,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:25,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,670 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:25,670 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:25,673 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,673 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,673 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,673 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:25,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:25,675 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7338743c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:25,675 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:25,676 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,677 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,677 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:25,677 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:25,677 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:25,677 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,677 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:25,678 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:25,678 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:25,678 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,678 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:25,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:25,680 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:25,680 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:25,680 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:25,681 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:25,681 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:25,683 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,683 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,683 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:25,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:25,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:25,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,685 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,687 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,690 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,690 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,690 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:25,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:25,690 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:25,690 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:25,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:25,690 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:25,690 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:25,690 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:25,690 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,690 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,690 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,690 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:25,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:25,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838925716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:25,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838925718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:25,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838925720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:25,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838925721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,723 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:25,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838925721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,772 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326c47f93f178bc49f0af7a835d315702db to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326c47f93f178bc49f0af7a835d315702db 2023-03-26 13:54:25,790 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:25,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:25,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:25,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:25,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:25,795 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:25,795 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:25,795 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:25,795 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:25,796 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,796 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,797 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,797 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:25,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:25,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:25,798 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,798 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:25,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:25,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:25,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:25,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:25,799 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:25,799 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:25,799 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:25,799 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:25,799 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:25,799 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:25,800 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:25,800 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:25,800 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:25,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:25,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:25,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:25,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:25,802 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,802 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:25,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:25,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16d48c42[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:25,806 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:25,806 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,808 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:25,809 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:25,809 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,809 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:25,810 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,810 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:25,810 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:25,810 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:25,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:25,813 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:25,813 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:25,813 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:25,813 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:25,813 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:25,815 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,815 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:25,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,815 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,817 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,819 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:25,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:25,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:25,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:25,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:25,823 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:25,823 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:25,822 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:25,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:25,823 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:25,923 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:25,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:25,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:25,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:25,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:25,926 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:25,927 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:25,927 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:25,927 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:25,927 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,927 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,928 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:25,928 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:25,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,929 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,929 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:25,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:25,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,930 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:25,930 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:25,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:25,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:25,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:25,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:25,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:25,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:25,931 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:25,931 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:25,931 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:25,931 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:25,931 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:25,934 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,934 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:25,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@744db5ee[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:25,937 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:25,937 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,938 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,938 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,938 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:25,938 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:25,938 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:25,939 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:25,942 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:25,942 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:25,942 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,942 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:25,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:25,948 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:25,948 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:25,948 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:25,948 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:25,948 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:25,950 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,950 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:25,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,950 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:25,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:25,951 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,952 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:25,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,955 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:25,955 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,955 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:25,955 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,955 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:25,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:25,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:25,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:25,955 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:25,955 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:25,955 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,955 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:25,955 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:25,955 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:25,955 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:25,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:25,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:26,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838926021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838926022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838926026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,029 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838926027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,030 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838926027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,056 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:26,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:26,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:26,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:26,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:26,058 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:26,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:26,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:26,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:26,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:26,060 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:26,060 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:26,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:26,061 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:26,061 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:26,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:26,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:26,061 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:26,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:26,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:26,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:26,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:26,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:26,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:26,062 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:26,062 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:26,062 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:26,062 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:26,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:26,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:26,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:26,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:26,064 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,064 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:26,064 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,064 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:26,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:26,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:26,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:26,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e9a8045[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:26,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,067 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,067 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:26,067 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:26,067 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:26,067 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,067 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:26,068 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,068 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:26,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:26,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:26,068 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:26,068 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:26,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:26,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:26,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:26,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:26,070 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:26,071 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:26,071 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:26,071 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:26,071 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:26,072 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,072 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:26,072 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:26,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:26,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:26,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:26,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,074 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:26,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:26,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,077 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,077 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:26,077 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:26,077 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:26,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:26,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:26,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:26,077 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:26,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:26,077 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,077 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:26,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:26,077 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:26,077 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:26,077 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:26,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:26,178 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:26,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:26,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:26,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:26,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:26,181 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:26,181 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:26,181 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:26,181 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:26,182 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,182 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:26,183 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,183 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:26,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:26,183 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,183 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:26,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:26,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:26,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:26,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:26,184 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=377, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/0fd097c52e1b463b8a52149f98d3c3ff 2023-03-26 13:54:26,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:26,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:26,184 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:26,185 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:26,185 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:26,185 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:26,185 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:26,185 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:26,186 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:26,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:26,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:26,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:26,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eb3923e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:26,190 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:26,190 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,191 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,191 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:26,191 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:26,191 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:26,192 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,192 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:26,194 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,194 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:26,194 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:26,194 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:26,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:26,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:26,195 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/63db185ceb6441748a00f5bb6156b59e 2023-03-26 13:54:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:26,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:26,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:26,207 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:26,207 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:26,207 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:26,207 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:26,207 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:26,209 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,209 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:26,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:26,209 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:26,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:26,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:26,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,213 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:26,214 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:26,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,219 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,219 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:26,219 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:26,219 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:26,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:26,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:26,220 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:26,220 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:26,219 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:26,220 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:26,219 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:26,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:26,220 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,220 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,220 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:26,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:26,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:26,227 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/5143513770fe4afcbe5bfc26adc9dd71 2023-03-26 13:54:26,232 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/0fd097c52e1b463b8a52149f98d3c3ff as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/0fd097c52e1b463b8a52149f98d3c3ff 2023-03-26 13:54:26,238 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/0fd097c52e1b463b8a52149f98d3c3ff, entries=350, sequenceid=377, filesize=52.9 K 2023-03-26 13:54:26,240 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/63db185ceb6441748a00f5bb6156b59e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/63db185ceb6441748a00f5bb6156b59e 2023-03-26 13:54:26,244 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/63db185ceb6441748a00f5bb6156b59e, entries=150, sequenceid=377, filesize=11.9 K 2023-03-26 13:54:26,245 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/5143513770fe4afcbe5bfc26adc9dd71 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/5143513770fe4afcbe5bfc26adc9dd71 2023-03-26 13:54:26,249 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/5143513770fe4afcbe5bfc26adc9dd71, entries=150, sequenceid=377, filesize=11.9 K 2023-03-26 13:54:26,250 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=154.31 KB/158010 for a739eafaf8b68b3135ee013236515a80 in 900ms, sequenceid=377, compaction requested=true 2023-03-26 13:54:26,250 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:26,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:26,250 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:54:26,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:26,250 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:54:26,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:26,252 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 132966 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:54:26,252 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/A is initiating minor compaction (all files) 2023-03-26 13:54:26,252 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/A in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:26,252 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/7796cf7ae41244e58cb3781708073c42, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/a71a5ccb08f04594bb58e655e8dea68c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/fee5081a5dd6417a89378994cbbb62d3, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/0fd097c52e1b463b8a52149f98d3c3ff] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=129.8 K 2023-03-26 13:54:26,252 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:54:26,252 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/B is initiating minor compaction (all files) 2023-03-26 13:54:26,252 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/B in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:26,252 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/386d137d738542618e5ebfed757401fd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/193baf37023d459bb6bf637683b6469c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/c1cec8fed9ba49f784d264d5b2db7690, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/63db185ceb6441748a00f5bb6156b59e] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=48.2 K 2023-03-26 13:54:26,252 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 7796cf7ae41244e58cb3781708073c42, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679838861880 2023-03-26 13:54:26,254 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 386d137d738542618e5ebfed757401fd, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679838861880 2023-03-26 13:54:26,254 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting a71a5ccb08f04594bb58e655e8dea68c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1679838862552 2023-03-26 13:54:26,254 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 193baf37023d459bb6bf637683b6469c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1679838862552 2023-03-26 13:54:26,254 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting fee5081a5dd6417a89378994cbbb62d3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679838863185 2023-03-26 13:54:26,255 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting c1cec8fed9ba49f784d264d5b2db7690, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679838863185 2023-03-26 13:54:26,255 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 0fd097c52e1b463b8a52149f98d3c3ff, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1679838865337 2023-03-26 13:54:26,255 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 63db185ceb6441748a00f5bb6156b59e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1679838865341 2023-03-26 13:54:26,268 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#B#compaction#202 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:26,269 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#A#compaction#201 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:26,289 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/791aa25ad0084358a730ea359ac83beb as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/791aa25ad0084358a730ea359ac83beb 2023-03-26 13:54:26,295 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a739eafaf8b68b3135ee013236515a80/B of a739eafaf8b68b3135ee013236515a80 into 791aa25ad0084358a730ea359ac83beb(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:26,295 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:26,295 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/B, priority=12, startTime=1679838866250; duration=0sec 2023-03-26 13:54:26,295 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:26,295 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:54:26,297 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:54:26,297 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/C is initiating minor compaction (all files) 2023-03-26 13:54:26,297 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/C in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:26,297 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/2de3ca8d707e488a95b99b2d0a0a58d4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/b5203f851f80423ca53e3ce37bb7465c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/e636bcbb84f84556b105ca1f97e883cc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/5143513770fe4afcbe5bfc26adc9dd71] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=48.2 K 2023-03-26 13:54:26,298 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 2de3ca8d707e488a95b99b2d0a0a58d4, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=317, earliestPutTs=1679838861880 2023-03-26 13:54:26,298 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting b5203f851f80423ca53e3ce37bb7465c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=354, earliestPutTs=1679838862552 2023-03-26 13:54:26,298 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e636bcbb84f84556b105ca1f97e883cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=366, earliestPutTs=1679838863185 2023-03-26 13:54:26,299 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 5143513770fe4afcbe5bfc26adc9dd71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1679838865341 2023-03-26 13:54:26,308 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#C#compaction#203 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:26,320 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:26,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:26,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:26,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:26,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:26,323 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:26,323 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:26,323 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:26,323 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:26,324 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,324 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:26,328 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,328 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:26,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:26,329 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,329 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:26,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:26,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:26,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:26,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:26,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:26,330 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:26,330 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:26,330 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:26,330 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:26,330 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:26,330 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:26,330 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:26,330 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=154.31 KB heapSize=379.13 KB 2023-03-26 13:54:26,331 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:26,331 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:26,331 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:26,331 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:26,331 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:26,331 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:26,368 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303267c2eac9f3fbe4cf5bf707ed88c3d6ac0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267c2eac9f3fbe4cf5bf707ed88c3d6ac0 2023-03-26 13:54:26,376 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=404, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/32b9b27cc7774158b3045f1947134c4e 2023-03-26 13:54:26,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:26,530 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:26,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838926536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838926536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838926537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838926539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838926539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838926641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838926642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838926642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838926642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,649 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838926648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,718 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/96984235d85942f59bd2adb50beb705e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/96984235d85942f59bd2adb50beb705e 2023-03-26 13:54:26,723 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a739eafaf8b68b3135ee013236515a80/C of a739eafaf8b68b3135ee013236515a80 into 96984235d85942f59bd2adb50beb705e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:26,723 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:26,723 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/C, priority=12, startTime=1679838866250; duration=0sec 2023-03-26 13:54:26,723 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:26,836 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/01ab8f9b420a4b8c9380a4d1a0ebb505 2023-03-26 13:54:26,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838926844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838926844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838926845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,848 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=404 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/d72b3327c14e4809a75731a56cb1138f 2023-03-26 13:54:26,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838926850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,852 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/32b9b27cc7774158b3045f1947134c4e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/32b9b27cc7774158b3045f1947134c4e 2023-03-26 13:54:26,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:26,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838926851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,855 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/32b9b27cc7774158b3045f1947134c4e, entries=150, sequenceid=404, filesize=25.4 K 2023-03-26 13:54:26,856 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/01ab8f9b420a4b8c9380a4d1a0ebb505 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/01ab8f9b420a4b8c9380a4d1a0ebb505 2023-03-26 13:54:26,860 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/01ab8f9b420a4b8c9380a4d1a0ebb505, entries=150, sequenceid=404, filesize=11.9 K 2023-03-26 13:54:26,861 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/d72b3327c14e4809a75731a56cb1138f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/d72b3327c14e4809a75731a56cb1138f 2023-03-26 13:54:26,868 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/d72b3327c14e4809a75731a56cb1138f, entries=150, sequenceid=404, filesize=11.9 K 2023-03-26 13:54:26,869 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~379.08 KB/388176, currentSize=53.67 KB/54960 for a739eafaf8b68b3135ee013236515a80 in 539ms, sequenceid=404, compaction requested=false 2023-03-26 13:54:26,869 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:26,869 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:26,869 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:26,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:26,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:26,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:26,871 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,871 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:26,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:26,871 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,871 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:26,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:26,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:26,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cb7b136[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:26,873 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:26,873 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,875 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,876 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:26,876 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:26,876 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:26,876 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,876 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:26,877 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:26,877 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:26,877 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,877 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:26,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:26,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:26,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:26,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:26,879 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:26,879 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:26,879 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:26,880 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:26,880 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:26,882 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,882 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:26,882 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:26,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:26,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:26,883 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:26,884 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:26,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:26,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:26,886 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:26,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:26,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:26,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:26,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:26,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:26,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:26,886 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:26,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:26,886 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:26,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:26,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:26,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:26,986 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:26,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:26,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:26,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:26,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:26,991 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:26,992 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:26,992 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:26,992 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:26,992 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,992 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:26,993 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:26,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:26,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:26,993 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:26,994 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:26,994 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:26,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:26,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:26,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:26,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:26,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:26,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:26,995 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:26,995 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:26,995 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:26,995 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:26,995 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:26,995 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:26,995 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:26,995 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 13:54:26,996 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:26,996 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:26,996 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:26,996 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:26,996 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:26,996 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:27,019 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326ad58506847bf476baa7b2ac8203dfe04 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ad58506847bf476baa7b2ac8203dfe04 2023-03-26 13:54:27,025 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=416, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/6161d677b6bc43439eea5778a3cce27c 2023-03-26 13:54:27,103 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/493884f3c0594b47a02dc687d4c0d2ff as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/493884f3c0594b47a02dc687d4c0d2ff 2023-03-26 13:54:27,111 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a739eafaf8b68b3135ee013236515a80/A of a739eafaf8b68b3135ee013236515a80 into 493884f3c0594b47a02dc687d4c0d2ff(size=26.2 K), total size for store is 51.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:27,111 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:27,111 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/A, priority=12, startTime=1679838866250; duration=0sec 2023-03-26 13:54:27,111 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:27,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:27,151 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:27,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838927166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838927168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,172 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1679838867158, max timestamp=1679838867170Num uniques 859; 2023-03-26 13:54:27,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838927168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-03-26 13:54:27,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1679838867158, max timestamp=1679838867170Num uniques 859; 2023-03-26 13:54:27,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-03-26 13:54:27,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1679838867158, max timestamp=1679838867170Num uniques 859; 2023-03-26 13:54:27,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-03-26 13:54:27,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838927188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838927189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838927273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838927273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838927273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,299 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838927295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838927295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838927476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838927476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,479 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/25f5a355008b4349a36d2b901b1a011e 2023-03-26 13:54:27,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838927477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838927501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838927501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838927781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838927781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838927781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,805 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838927804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:27,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838927804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,890 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=416 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/30b2dfe9e28a477cb4090a534df0a0e0 2023-03-26 13:54:27,894 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/6161d677b6bc43439eea5778a3cce27c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/6161d677b6bc43439eea5778a3cce27c 2023-03-26 13:54:27,898 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/6161d677b6bc43439eea5778a3cce27c, entries=150, sequenceid=416, filesize=25.4 K 2023-03-26 13:54:27,898 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/25f5a355008b4349a36d2b901b1a011e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/25f5a355008b4349a36d2b901b1a011e 2023-03-26 13:54:27,902 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/25f5a355008b4349a36d2b901b1a011e, entries=150, sequenceid=416, filesize=11.9 K 2023-03-26 13:54:27,902 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/30b2dfe9e28a477cb4090a534df0a0e0 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/30b2dfe9e28a477cb4090a534df0a0e0 2023-03-26 13:54:27,907 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/30b2dfe9e28a477cb4090a534df0a0e0, entries=150, sequenceid=416, filesize=11.9 K 2023-03-26 13:54:27,908 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for a739eafaf8b68b3135ee013236515a80 in 913ms, sequenceid=416, compaction requested=true 2023-03-26 13:54:27,908 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:27,908 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:27,908 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:27,908 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:27,908 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,908 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:27,908 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:27,910 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:27,910 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:27,910 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:27,910 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:27,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:27,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:27,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:27,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:27,911 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:27,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65cc898b[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:27,911 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:27,912 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:27,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:27,912 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:27,913 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:27,913 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:27,913 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,913 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:27,914 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,914 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:27,914 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:27,914 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:27,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:27,916 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:27,917 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:27,917 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:27,917 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:27,917 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:27,919 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:27,919 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:27,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:27,919 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:27,920 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:27,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,923 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,923 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:27,923 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:27,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:27,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:27,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:27,923 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:27,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:27,923 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:27,923 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:27,924 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:27,924 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:27,924 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:27,924 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:27,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:27,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:27,924 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:27,924 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,024 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:28,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:28,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:28,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:28,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:28,027 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:28,027 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:28,027 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:28,027 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:28,027 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,027 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,028 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,028 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:28,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:28,029 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,029 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:28,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:28,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:28,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:28,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:28,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:28,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:28,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:28,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:28,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:28,031 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:28,031 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:28,031 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=134.18 KB heapSize=356.86 KB 2023-03-26 13:54:28,031 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:28,031 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:28,031 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:28,031 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:28,031 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:28,031 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:28,044 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326c960814eee4644e890812d99466736b3 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326c960814eee4644e890812d99466736b3 2023-03-26 13:54:28,074 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=448, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/be7d1790fe474393a2180ff2e4080807 2023-03-26 13:54:28,089 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/a3f20253f34b446b89665ee0f6a36b70 2023-03-26 13:54:28,101 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=448 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/798cff5240c346f3a353c24ca71db73f 2023-03-26 13:54:28,117 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/be7d1790fe474393a2180ff2e4080807 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/be7d1790fe474393a2180ff2e4080807 2023-03-26 13:54:28,121 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/be7d1790fe474393a2180ff2e4080807, entries=150, sequenceid=448, filesize=25.4 K 2023-03-26 13:54:28,122 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/a3f20253f34b446b89665ee0f6a36b70 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a3f20253f34b446b89665ee0f6a36b70 2023-03-26 13:54:28,126 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a3f20253f34b446b89665ee0f6a36b70, entries=150, sequenceid=448, filesize=11.9 K 2023-03-26 13:54:28,127 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/798cff5240c346f3a353c24ca71db73f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/798cff5240c346f3a353c24ca71db73f 2023-03-26 13:54:28,140 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/798cff5240c346f3a353c24ca71db73f, entries=150, sequenceid=448, filesize=11.9 K 2023-03-26 13:54:28,141 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~356.81 KB/365376, currentSize=0 B/0 for a739eafaf8b68b3135ee013236515a80 in 110ms, sequenceid=448, compaction requested=true 2023-03-26 13:54:28,141 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:28,141 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:28,141 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:28,141 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:28,141 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,141 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:28,141 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:28,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,143 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:28,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:28,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f7eb5df[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:28,145 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:28,145 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,149 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,149 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:28,150 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:28,150 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:28,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:28,151 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,151 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:28,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,151 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:28,151 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:28,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:28,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:28,154 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:28,154 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:28,154 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:28,154 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:28,154 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:28,155 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,155 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,155 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:28,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,157 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,159 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,159 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,159 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:28,159 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,159 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:28,160 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:28,160 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:28,160 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:28,160 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,160 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,160 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:28,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:28,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:28,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:28,161 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:28,161 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:28,261 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:28,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:28,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:28,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:28,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:28,265 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:28,265 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:28,265 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:28,265 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:28,266 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,266 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,270 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:28,270 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,272 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,272 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:28,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:28,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:28,272 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:28,272 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:28,272 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:28,272 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:28,272 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:28,272 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:28,272 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:28,273 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:28,273 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:28,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:28,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:28,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:28,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:28,274 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,274 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,275 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,275 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:28,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d8efc6c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:28,277 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:28,277 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,278 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,278 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:28,278 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:28,278 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:28,278 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,278 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:28,279 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:28,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,279 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:28,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,280 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:28,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:28,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:28,283 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:28,283 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:28,283 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:28,283 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:28,283 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:28,286 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,286 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,286 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:28,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:28,287 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,289 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,291 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:28,291 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:28,291 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:28,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:28,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:28,292 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:28,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:28,291 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:28,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:28,292 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:28,292 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:28,292 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,292 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,292 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,293 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,293 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:28,332 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 13:54:28,332 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:28,332 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:28,332 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:28,333 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:28,333 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:28,333 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:28,359 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326ed338cb8a96f471d88d032976bad7feb to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ed338cb8a96f471d88d032976bad7feb 2023-03-26 13:54:28,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679838868335, max timestamp=1679838868363Num uniques 859; 2023-03-26 13:54:28,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679838868335, max timestamp=1679838868363Num uniques 859; 2023-03-26 13:54:28,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-26 13:54:28,365 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-26 13:54:28,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679838868335, max timestamp=1679838868363Num uniques 859; 2023-03-26 13:54:28,366 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-26 13:54:28,376 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=460, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/87cc6fe787284eb288853e0ecc17241a 2023-03-26 13:54:28,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:40056 deadline: 1679838928367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838928368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838928372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838928377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838928377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,392 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:28,393 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/4e4e3861313047f48709e75898e3d15e 2023-03-26 13:54:28,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:28,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:28,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:28,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:28,397 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:28,397 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:28,397 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:28,397 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:28,398 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,398 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,400 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:28,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:28,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:28,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,401 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,401 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:28,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:28,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:28,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:28,403 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:28,403 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:28,403 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:28,408 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:28,408 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:28,408 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:28,408 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:28,409 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:28,409 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:28,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:28,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:28,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,412 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:28,412 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:28,414 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,414 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,414 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,414 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:28,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:28,417 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cf1d0ea[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:28,417 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:28,417 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,420 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,420 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:28,420 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:28,420 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:28,421 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,421 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:28,422 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,423 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:28,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:28,424 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:28,424 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:28,424 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:28,424 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:28,424 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:28,426 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,426 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,426 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:28,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:28,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:28,426 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,428 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,431 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/2483557bf34e433cbb388bd8ca10fd1b 2023-03-26 13:54:28,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,432 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,432 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,432 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,432 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:28,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:28,432 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,432 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,432 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:28,432 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,432 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:28,433 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:28,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:28,433 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:28,433 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:28,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:28,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:28,433 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:28,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/87cc6fe787284eb288853e0ecc17241a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/87cc6fe787284eb288853e0ecc17241a 2023-03-26 13:54:28,444 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/87cc6fe787284eb288853e0ecc17241a, entries=300, sequenceid=460, filesize=46.1 K 2023-03-26 13:54:28,445 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/4e4e3861313047f48709e75898e3d15e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/4e4e3861313047f48709e75898e3d15e 2023-03-26 13:54:28,453 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/4e4e3861313047f48709e75898e3d15e, entries=300, sequenceid=460, filesize=19.0 K 2023-03-26 13:54:28,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/2483557bf34e433cbb388bd8ca10fd1b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/2483557bf34e433cbb388bd8ca10fd1b 2023-03-26 13:54:28,458 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/2483557bf34e433cbb388bd8ca10fd1b, entries=150, sequenceid=460, filesize=11.9 K 2023-03-26 13:54:28,459 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=154.31 KB/158010 for a739eafaf8b68b3135ee013236515a80 in 127ms, sequenceid=460, compaction requested=true 2023-03-26 13:54:28,459 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:28,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:28,460 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 13:54:28,460 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 13:54:28,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:28,460 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:28,461 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 152104 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 13:54:28,462 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/A is initiating minor compaction (all files) 2023-03-26 13:54:28,462 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/A in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:28,462 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/493884f3c0594b47a02dc687d4c0d2ff, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/32b9b27cc7774158b3045f1947134c4e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/6161d677b6bc43439eea5778a3cce27c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/be7d1790fe474393a2180ff2e4080807, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/87cc6fe787284eb288853e0ecc17241a] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=148.5 K 2023-03-26 13:54:28,462 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 69000 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 13:54:28,462 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/B is initiating minor compaction (all files) 2023-03-26 13:54:28,462 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/B in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:28,462 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/791aa25ad0084358a730ea359ac83beb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/01ab8f9b420a4b8c9380a4d1a0ebb505, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/25f5a355008b4349a36d2b901b1a011e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a3f20253f34b446b89665ee0f6a36b70, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/4e4e3861313047f48709e75898e3d15e] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=67.4 K 2023-03-26 13:54:28,463 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 493884f3c0594b47a02dc687d4c0d2ff, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1679838865341 2023-03-26 13:54:28,463 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 791aa25ad0084358a730ea359ac83beb, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1679838865341 2023-03-26 13:54:28,463 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 32b9b27cc7774158b3045f1947134c4e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1679838865387 2023-03-26 13:54:28,463 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 01ab8f9b420a4b8c9380a4d1a0ebb505, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1679838865387 2023-03-26 13:54:28,464 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 6161d677b6bc43439eea5778a3cce27c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1679838866530 2023-03-26 13:54:28,464 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 25f5a355008b4349a36d2b901b1a011e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1679838866530 2023-03-26 13:54:28,464 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting be7d1790fe474393a2180ff2e4080807, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1679838867185 2023-03-26 13:54:28,465 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting a3f20253f34b446b89665ee0f6a36b70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1679838867185 2023-03-26 13:54:28,465 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 87cc6fe787284eb288853e0ecc17241a, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1679838868301 2023-03-26 13:54:28,465 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 4e4e3861313047f48709e75898e3d15e, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1679838868301 2023-03-26 13:54:28,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:28,483 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=161.02 KB heapSize=399.05 KB 2023-03-26 13:54:28,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:28,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:28,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:28,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:28,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:28,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:28,486 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#B#compaction#217 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:28,492 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838928488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838928489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40056 deadline: 1679838928491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838928492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838928490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,521 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#A#compaction#216 average throughput is 0.48 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:28,533 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:28,534 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/bcffdad9d36c4d6cbbcb6f8456a90325 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/bcffdad9d36c4d6cbbcb6f8456a90325 2023-03-26 13:54:28,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:28,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:28,541 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a739eafaf8b68b3135ee013236515a80/B of a739eafaf8b68b3135ee013236515a80 into bcffdad9d36c4d6cbbcb6f8456a90325(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:28,541 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:28,541 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/B, priority=11, startTime=1679838868460; duration=0sec 2023-03-26 13:54:28,541 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:28,541 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 13:54:28,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:28,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:28,544 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:28,544 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:28,544 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:28,544 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:28,544 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,544 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,545 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 13:54:28,545 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/C is initiating minor compaction (all files) 2023-03-26 13:54:28,545 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/C in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:28,545 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/96984235d85942f59bd2adb50beb705e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/d72b3327c14e4809a75731a56cb1138f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/30b2dfe9e28a477cb4090a534df0a0e0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/798cff5240c346f3a353c24ca71db73f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/2483557bf34e433cbb388bd8ca10fd1b] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=60.2 K 2023-03-26 13:54:28,545 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 96984235d85942f59bd2adb50beb705e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1679838865341 2023-03-26 13:54:28,546 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting d72b3327c14e4809a75731a56cb1138f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=404, earliestPutTs=1679838865387 2023-03-26 13:54:28,546 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 30b2dfe9e28a477cb4090a534df0a0e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=416, earliestPutTs=1679838866530 2023-03-26 13:54:28,547 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 798cff5240c346f3a353c24ca71db73f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=448, earliestPutTs=1679838867185 2023-03-26 13:54:28,547 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:28,547 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:28,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:28,547 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 2483557bf34e433cbb388bd8ca10fd1b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1679838868328 2023-03-26 13:54:28,547 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,548 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:28,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:28,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:28,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:28,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:28,557 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:28,557 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:28,557 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:28,564 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:28,564 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:28,564 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:28,564 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:28,564 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:28,564 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:28,564 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:28,564 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:28,564 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,564 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:28,564 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:28,566 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,566 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,566 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,566 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:28,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d709a2e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:28,568 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:28,568 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,569 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:28,570 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:28,570 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:28,570 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,570 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:28,571 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:28,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:28,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:28,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:28,575 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:28,575 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:28,575 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:28,575 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:28,575 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:28,578 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#C#compaction#219 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:28,583 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,583 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:28,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,583 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,584 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,588 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,588 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,588 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,588 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,588 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:28,588 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:28,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:28,588 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,588 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:28,588 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:28,588 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:28,589 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:28,589 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:28,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:28,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:28,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:28,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838928594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,604 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/5a2e190e508b4d02a56a4574103cf7b2 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/5a2e190e508b4d02a56a4574103cf7b2 2023-03-26 13:54:28,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838928600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40056 deadline: 1679838928600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838928600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838928601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,611 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a739eafaf8b68b3135ee013236515a80/C of a739eafaf8b68b3135ee013236515a80 into 5a2e190e508b4d02a56a4574103cf7b2(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:28,611 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:28,611 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/C, priority=11, startTime=1679838868460; duration=0sec 2023-03-26 13:54:28,611 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:28,689 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:28,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:28,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:28,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:28,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:28,702 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:28,702 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:28,702 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:28,702 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:28,702 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,702 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,705 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:28,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:28,705 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,706 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,706 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:28,706 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:28,707 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:28,707 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:28,707 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:28,707 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:28,708 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:28,708 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:28,708 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:28,708 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:28,708 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:28,708 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:28,708 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:28,708 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:28,708 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:28,708 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,708 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:28,708 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:28,713 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,713 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,714 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,714 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:28,715 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:28,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@671a1c12[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:28,715 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,718 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:28,718 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:28,719 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,719 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:28,720 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,720 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:28,720 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:28,720 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:28,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:28,723 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:28,723 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:28,723 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:28,723 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:28,723 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:28,726 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,726 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:28,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,726 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,727 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,730 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,734 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,734 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,734 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,734 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,734 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:28,734 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:28,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:28,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:28,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:28,734 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:28,734 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,734 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,734 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:28,734 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:28,734 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:28,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:28,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:28,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838928800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838928805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40056 deadline: 1679838928805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838928806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:28,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838928807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,834 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:28,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:28,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:28,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:28,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:28,839 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:28,839 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:28,839 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:28,839 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:28,839 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,839 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,841 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:28,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:28,841 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,842 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,842 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:28,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:28,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:28,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:28,843 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:28,843 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:28,843 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:28,844 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:28,844 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:28,844 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:28,844 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:28,844 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:28,844 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:28,844 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:28,844 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:28,844 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,844 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:28,844 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:28,846 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,846 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,846 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,846 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:28,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d9de101[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:28,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:28,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,851 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,851 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:28,851 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:28,851 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:28,854 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,854 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:28,854 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:28,854 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:28,854 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:28,854 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:28,854 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:28,854 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:28,854 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:28,855 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:28,855 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:28,856 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,856 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:28,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:28,857 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,857 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,858 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,860 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,860 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,860 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:28,860 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:28,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:28,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:28,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:28,860 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,861 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,861 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,861 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,861 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:28,861 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:28,861 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:28,861 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:28,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:28,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:28,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:28,941 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032686295bb267b345f9a201fae295de1c93 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032686295bb267b345f9a201fae295de1c93 2023-03-26 13:54:28,947 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=487, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/87a5e52f570f493da3c8b59736f12661 2023-03-26 13:54:28,952 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/5099275104444f77b3dbef38fddf9952 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/5099275104444f77b3dbef38fddf9952 2023-03-26 13:54:28,961 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:28,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:28,963 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a739eafaf8b68b3135ee013236515a80/A of a739eafaf8b68b3135ee013236515a80 into 5099275104444f77b3dbef38fddf9952(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:28,963 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:28,963 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/A, priority=11, startTime=1679838868459; duration=0sec 2023-03-26 13:54:28,963 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:28,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:28,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:28,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:28,968 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:28,968 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:28,968 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:28,968 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:28,969 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,969 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,970 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,970 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:28,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:28,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:28,970 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,970 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:28,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,972 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/75a20c01eab44cf1a8a0685f32206a9b 2023-03-26 13:54:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:28,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,973 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:28,973 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:28,973 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:28,973 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:28,973 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:28,974 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:28,974 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:28,974 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:28,974 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:28,974 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:28,975 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:28,975 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:28,975 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:28,975 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:28,975 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,975 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:28,975 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:28,978 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,978 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,978 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,978 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fb906d8[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:28,981 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:28,981 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,982 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,982 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:28,982 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:28,982 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:28,982 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,982 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:28,984 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:28,984 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:28,984 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,984 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:28,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:28,988 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:28,988 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:28,988 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:28,988 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:28,988 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:28,991 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,991 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:28,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,991 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:28,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:28,992 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,993 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,997 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:28,997 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:28,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:28,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:28,997 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:28,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:28,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:28,997 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:28,997 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:28,998 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,998 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,998 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:28,998 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:28,998 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,998 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:28,998 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:28,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:28,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:29,016 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/64de000ed33f4405b6a52ab30d903f15 2023-03-26 13:54:29,021 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/87a5e52f570f493da3c8b59736f12661 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/87a5e52f570f493da3c8b59736f12661 2023-03-26 13:54:29,027 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/87a5e52f570f493da3c8b59736f12661, entries=200, sequenceid=487, filesize=32.3 K 2023-03-26 13:54:29,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/75a20c01eab44cf1a8a0685f32206a9b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/75a20c01eab44cf1a8a0685f32206a9b 2023-03-26 13:54:29,033 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/75a20c01eab44cf1a8a0685f32206a9b, entries=150, sequenceid=487, filesize=11.9 K 2023-03-26 13:54:29,035 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/64de000ed33f4405b6a52ab30d903f15 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/64de000ed33f4405b6a52ab30d903f15 2023-03-26 13:54:29,045 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/64de000ed33f4405b6a52ab30d903f15, entries=150, sequenceid=487, filesize=11.9 K 2023-03-26 13:54:29,046 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~399 KB/408576, currentSize=40.25 KB/41220 for a739eafaf8b68b3135ee013236515a80 in 563ms, sequenceid=487, compaction requested=false 2023-03-26 13:54:29,046 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:29,097 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:29,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:29,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:29,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:29,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:29,101 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:29,101 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:29,101 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:29,101 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:29,101 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,101 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,103 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,103 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:29,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:29,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:29,103 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:54:29,103 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,103 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:29,103 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:29,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,103 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:29,103 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:29,103 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:29,103 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:29,104 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:29,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:29,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:29,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:29,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:29,104 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:29,104 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:29,104 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:29,105 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:29,105 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:29,105 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:29,105 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:29,105 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:29,108 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:29,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:29,108 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,108 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,109 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:29,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:29,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:29,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:29,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64156eb[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:29,111 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:29,111 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,112 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,112 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:29,112 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:29,112 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:29,112 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,112 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:29,118 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303262037996c9ff4438989c67f74ef383b91 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303262037996c9ff4438989c67f74ef383b91 2023-03-26 13:54:29,119 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:29,119 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:29,119 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,119 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:29,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:29,123 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:29,123 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:29,124 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:29,124 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:29,124 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:29,125 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=500, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/5a31f877a04746bb8fd630099e7acd79 2023-03-26 13:54:29,126 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,126 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:29,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,126 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:29,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:29,127 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,128 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,132 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:29,132 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,132 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:29,132 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:29,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:29,132 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:29,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:29,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:29,132 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,132 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,132 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,132 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,132 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,132 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:29,133 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:29,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:29,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1679838869120, max timestamp=1679838869131Num uniques 859; 2023-03-26 13:54:29,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1679838869120, max timestamp=1679838869131Num uniques 859; 2023-03-26 13:54:29,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1679838869120, max timestamp=1679838869131Num uniques 859; 2023-03-26 13:54:29,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-26 13:54:29,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-26 13:54:29,134 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-26 13:54:29,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838929143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838929145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838929146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838929147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,149 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40056 deadline: 1679838929147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,233 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:29,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:29,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:29,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:29,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:29,236 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:29,236 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:29,236 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:29,236 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:29,236 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,236 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,238 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:29,238 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:29,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:29,238 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,238 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:29,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:29,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:29,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:29,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:29,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:29,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:29,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:29,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:29,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:29,242 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:29,242 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:29,242 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:29,242 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:29,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:29,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:29,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:29,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:29,245 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:29,245 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:29,245 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,246 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:29,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:29,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:29,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:29,247 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:29,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@310c30bd[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:29,247 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,249 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:29,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:29,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:29,249 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,249 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:29,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838929249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:29,251 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,251 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:29,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:29,255 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:29,255 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:29,255 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:29,255 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:29,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,255 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:29,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838929250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838929254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40056 deadline: 1679838929254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838929254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,258 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,258 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:29,258 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:29,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,260 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,266 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,266 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,266 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:29,266 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:29,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:29,266 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,266 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,266 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,266 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,266 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:29,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:29,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:29,266 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:29,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:29,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:29,266 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:29,266 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:29,367 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:29,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:29,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:29,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:29,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:29,370 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:29,370 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:29,370 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:29,370 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:29,371 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,371 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,372 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:29,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:29,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:29,372 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,373 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,373 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:29,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:29,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:29,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:29,374 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:29,374 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:29,374 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:29,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:29,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:29,376 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:29,376 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:29,376 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:29,376 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:29,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:29,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:29,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:29,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:29,378 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:29,378 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:29,378 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,378 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:29,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:29,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:29,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53fb4a74[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:29,380 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:29,380 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,381 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,381 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:29,381 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,381 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:29,381 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:29,381 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:29,383 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,383 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,383 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:29,383 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:29,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:29,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:29,386 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:29,386 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:29,386 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:29,386 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:29,386 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:29,388 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,388 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:29,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,389 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:29,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,399 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,401 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,401 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,402 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,402 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:29,402 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:29,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:29,402 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,402 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:29,402 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:29,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:29,402 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:29,402 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,402 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:29,402 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:29,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838929451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,456 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838929456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838929458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838929458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,462 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40056 deadline: 1679838929459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,502 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:29,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:29,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:29,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:29,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:29,505 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:29,505 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:29,505 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:29,505 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:29,506 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,506 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,507 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,507 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:29,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:29,507 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,507 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:29,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:29,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:29,508 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:29,508 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:29,508 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:29,509 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:29,509 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:29,509 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:29,509 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:29,509 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:29,509 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:29,509 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:29,509 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:29,509 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,509 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:29,509 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:29,514 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,515 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:29,515 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,515 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:29,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:29,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44630915[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:29,516 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:29,516 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,517 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,517 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:29,517 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:29,517 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:29,517 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,517 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:29,518 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:29,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:29,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:29,518 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:29,519 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:29,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:29,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:29,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:29,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:29,522 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:29,522 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:29,522 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:29,522 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:29,522 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:29,523 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,523 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:29,523 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,524 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:29,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,525 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,531 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,531 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,531 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,531 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,531 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:29,531 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:29,531 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:29,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:29,531 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:29,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:29,531 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:29,532 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,532 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:29,532 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:29,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:29,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:29,542 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/bce56b702c074788819b2e60e72df575 2023-03-26 13:54:29,558 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=500 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/7c02a820de684cd99bad0351431783c9 2023-03-26 13:54:29,566 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/5a31f877a04746bb8fd630099e7acd79 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/5a31f877a04746bb8fd630099e7acd79 2023-03-26 13:54:29,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/5a31f877a04746bb8fd630099e7acd79, entries=200, sequenceid=500, filesize=32.3 K 2023-03-26 13:54:29,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/bce56b702c074788819b2e60e72df575 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/bce56b702c074788819b2e60e72df575 2023-03-26 13:54:29,577 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/bce56b702c074788819b2e60e72df575, entries=150, sequenceid=500, filesize=11.9 K 2023-03-26 13:54:29,578 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/7c02a820de684cd99bad0351431783c9 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/7c02a820de684cd99bad0351431783c9 2023-03-26 13:54:29,582 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/7c02a820de684cd99bad0351431783c9, entries=150, sequenceid=500, filesize=11.9 K 2023-03-26 13:54:29,583 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for a739eafaf8b68b3135ee013236515a80 in 480ms, sequenceid=500, compaction requested=true 2023-03-26 13:54:29,583 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:29,583 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:29,583 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:29,583 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:29,583 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:29,584 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:29,584 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 93138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:29,584 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/A is initiating minor compaction (all files) 2023-03-26 13:54:29,584 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:29,585 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/A in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:29,585 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/B is initiating minor compaction (all files) 2023-03-26 13:54:29,585 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/5099275104444f77b3dbef38fddf9952, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/87a5e52f570f493da3c8b59736f12661, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/5a31f877a04746bb8fd630099e7acd79] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=91.0 K 2023-03-26 13:54:29,585 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/B in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:29,585 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/bcffdad9d36c4d6cbbcb6f8456a90325, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/75a20c01eab44cf1a8a0685f32206a9b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/bce56b702c074788819b2e60e72df575] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=36.6 K 2023-03-26 13:54:29,585 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 5099275104444f77b3dbef38fddf9952, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1679838868328 2023-03-26 13:54:29,585 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting bcffdad9d36c4d6cbbcb6f8456a90325, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1679838868328 2023-03-26 13:54:29,585 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 87a5e52f570f493da3c8b59736f12661, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1679838868370 2023-03-26 13:54:29,585 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 75a20c01eab44cf1a8a0685f32206a9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1679838868375 2023-03-26 13:54:29,586 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 5a31f877a04746bb8fd630099e7acd79, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1679838868488 2023-03-26 13:54:29,586 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting bce56b702c074788819b2e60e72df575, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1679838868489 2023-03-26 13:54:29,592 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#B#compaction#226 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:29,597 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#A#compaction#225 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:29,613 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/90646c01d4a544188595d8d69a9528e4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/90646c01d4a544188595d8d69a9528e4 2023-03-26 13:54:29,617 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a739eafaf8b68b3135ee013236515a80/A of a739eafaf8b68b3135ee013236515a80 into 90646c01d4a544188595d8d69a9528e4(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:29,617 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:29,617 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/A, priority=13, startTime=1679838869583; duration=0sec 2023-03-26 13:54:29,617 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:29,617 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:29,618 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:29,618 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a739eafaf8b68b3135ee013236515a80/C is initiating minor compaction (all files) 2023-03-26 13:54:29,618 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a739eafaf8b68b3135ee013236515a80/C in TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:29,618 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/5a2e190e508b4d02a56a4574103cf7b2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/64de000ed33f4405b6a52ab30d903f15, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/7c02a820de684cd99bad0351431783c9] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp, totalSize=36.6 K 2023-03-26 13:54:29,619 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 5a2e190e508b4d02a56a4574103cf7b2, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1679838868328 2023-03-26 13:54:29,619 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 64de000ed33f4405b6a52ab30d903f15, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1679838868375 2023-03-26 13:54:29,619 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 7c02a820de684cd99bad0351431783c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=500, earliestPutTs=1679838868489 2023-03-26 13:54:29,627 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a739eafaf8b68b3135ee013236515a80#C#compaction#227 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:29,632 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:29,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:29,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:29,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:29,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:29,638 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:29,638 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:29,638 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:29,638 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:29,638 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,639 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,640 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:29,640 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:29,640 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,640 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:29,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:29,641 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:29,641 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:29,641 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:29,641 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:29,641 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:29,641 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:29,641 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:29,641 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:29,641 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:29,641 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:29,642 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=140.89 KB heapSize=381.47 KB 2023-03-26 13:54:29,642 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:29,642 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:29,642 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:29,642 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:29,642 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:29,642 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:29,664 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303263a0e15d290404db69d56b9cf5e7c802a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263a0e15d290404db69d56b9cf5e7c802a 2023-03-26 13:54:29,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:29,756 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:29,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,769 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838929762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838929763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40056 deadline: 1679838929763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838929769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838929769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838929870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838929874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:29,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:29,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838929874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,003 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/0fc99758c7d6444abf0575b74fc61135 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/0fc99758c7d6444abf0575b74fc61135 2023-03-26 13:54:30,007 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a739eafaf8b68b3135ee013236515a80/B of a739eafaf8b68b3135ee013236515a80 into 0fc99758c7d6444abf0575b74fc61135(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:30,008 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:30,008 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/B, priority=13, startTime=1679838869583; duration=0sec 2023-03-26 13:54:30,008 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:30,037 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/2b1b7dd7bd10434d831e3f2385703975 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/2b1b7dd7bd10434d831e3f2385703975 2023-03-26 13:54:30,042 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a739eafaf8b68b3135ee013236515a80/C of a739eafaf8b68b3135ee013236515a80 into 2b1b7dd7bd10434d831e3f2385703975(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:30,042 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:30,042 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80., storeName=a739eafaf8b68b3135ee013236515a80/C, priority=13, startTime=1679838869584; duration=0sec 2023-03-26 13:54:30,042 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:30,072 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=534, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/adab2b725fee4664aa8e2b8bc86abea8 2023-03-26 13:54:30,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:30,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838930076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:30,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838930077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,082 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/49f3f0bc2b25418198a624d7cc9b397b 2023-03-26 13:54:30,085 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:30,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838930085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,097 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/4a7677d885904fe1ba97aadb31012867 2023-03-26 13:54:30,101 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/adab2b725fee4664aa8e2b8bc86abea8 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/adab2b725fee4664aa8e2b8bc86abea8 2023-03-26 13:54:30,104 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/adab2b725fee4664aa8e2b8bc86abea8, entries=150, sequenceid=534, filesize=25.4 K 2023-03-26 13:54:30,105 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/49f3f0bc2b25418198a624d7cc9b397b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/49f3f0bc2b25418198a624d7cc9b397b 2023-03-26 13:54:30,108 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/49f3f0bc2b25418198a624d7cc9b397b, entries=150, sequenceid=534, filesize=11.9 K 2023-03-26 13:54:30,109 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/4a7677d885904fe1ba97aadb31012867 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/4a7677d885904fe1ba97aadb31012867 2023-03-26 13:54:30,113 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/4a7677d885904fe1ba97aadb31012867, entries=150, sequenceid=534, filesize=11.9 K 2023-03-26 13:54:30,116 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~381.42 KB/390576, currentSize=46.96 KB/48090 for a739eafaf8b68b3135ee013236515a80 in 474ms, sequenceid=534, compaction requested=false 2023-03-26 13:54:30,116 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:30,116 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:30,116 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:30,116 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:30,116 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,116 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:30,116 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:30,118 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:30,118 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:30,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:30,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4819f97c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:30,121 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:30,121 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:30,123 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:30,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:30,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:30,123 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:30,123 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:30,123 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:30,123 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,123 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:30,126 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:30,126 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:30,126 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:30,126 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:30,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:30,128 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:30,129 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:30,129 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:30,129 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:30,129 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:30,130 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:30,130 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:30,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:30,130 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:30,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:30,131 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:30,132 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:30,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,134 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:30,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:30,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:30,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:30,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:30,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:30,135 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:30,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:30,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:30,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:30,135 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:30,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:30,135 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:30,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:30,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:30,235 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:30,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:30,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:30,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:30,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:30,239 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:30,239 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:30,239 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:30,239 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:30,239 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:30,239 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:30,240 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:30,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:30,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:30,240 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:30,241 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,241 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:30,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:30,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:30,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:30,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:30,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:30,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:30,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:30,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:30,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:30,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:30,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:30,242 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:30,242 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:30,242 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:54:30,242 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:30,242 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:30,242 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:30,243 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:30,243 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:30,243 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:30,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:30,273 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:30,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679838870288, max timestamp=1679838870296Num uniques 859; 2023-03-26 13:54:30,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 13:54:30,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679838870288, max timestamp=1679838870296Num uniques 859; 2023-03-26 13:54:30,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679838870288, max timestamp=1679838870296Num uniques 859; 2023-03-26 13:54:30,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 13:54:30,298 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 13:54:30,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:30,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838930331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:30,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40056 deadline: 1679838930335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:30,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838930380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:30,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838930384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,388 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:30,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838930387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:30,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838930438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:30,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40056 deadline: 1679838930439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,640 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:30,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838930639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:30,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40056 deadline: 1679838930641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,658 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032693b23e7fc1774e1482060b37fc8f352e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032693b23e7fc1774e1482060b37fc8f352e 2023-03-26 13:54:30,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:30,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838930885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:30,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838930886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:30,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838930892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:30,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838930942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:30,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:30,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40056 deadline: 1679838930945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,063 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=546, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/821b5ffd1a05423483741a0df84ba9ea 2023-03-26 13:54:31,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:31,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40054 deadline: 1679838931446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:31,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40056 deadline: 1679838931446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,474 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/2e74822727b64c878a0bc99d4914a575 2023-03-26 13:54:31,885 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/a3deccec3dc84a1293a635fc98dafa7a 2023-03-26 13:54:31,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:31,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40020 deadline: 1679838931888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,890 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/821b5ffd1a05423483741a0df84ba9ea as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/821b5ffd1a05423483741a0df84ba9ea 2023-03-26 13:54:31,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:31,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40040 deadline: 1679838931890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,895 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/821b5ffd1a05423483741a0df84ba9ea, entries=150, sequenceid=546, filesize=25.4 K 2023-03-26 13:54:31,896 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/2e74822727b64c878a0bc99d4914a575 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/2e74822727b64c878a0bc99d4914a575 2023-03-26 13:54:31,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:31,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:40010 deadline: 1679838931895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,900 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/2e74822727b64c878a0bc99d4914a575, entries=150, sequenceid=546, filesize=11.9 K 2023-03-26 13:54:31,901 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/a3deccec3dc84a1293a635fc98dafa7a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/a3deccec3dc84a1293a635fc98dafa7a 2023-03-26 13:54:31,909 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/a3deccec3dc84a1293a635fc98dafa7a, entries=150, sequenceid=546, filesize=11.9 K 2023-03-26 13:54:31,910 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for a739eafaf8b68b3135ee013236515a80 in 1668ms, sequenceid=546, compaction requested=true 2023-03-26 13:54:31,910 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:31,910 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:31,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,913 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:31,913 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:31,913 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:31,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:31,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:31,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:31,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:31,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18c0b11a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:31,915 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:31,915 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:31,916 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:31,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:31,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:31,916 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:31,916 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:31,916 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:31,917 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,918 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:31,918 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:31,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:31,918 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:31,919 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:31,919 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:31,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:31,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:31,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:31,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:31,921 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:31,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:31,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:31,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:31,921 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:31,923 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:31,923 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:31,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:31,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:31,923 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:31,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:31,925 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:31,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:31,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:31,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:31,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,926 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:31,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:31,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,930 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,930 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:31,930 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:31,930 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:31,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:31,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:31,930 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:31,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:31,930 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:31,931 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:31,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:31,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:31,931 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:31,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:31,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:31,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:32,031 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:32,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:32,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:32,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:32,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:32,034 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:32,034 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:32,035 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:32,035 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:32,035 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:32,035 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:32,036 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:32,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:32,036 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:32,037 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:32,037 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:32,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:32,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:32,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:32,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:32,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:32,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:32,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:32,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:32,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:32,038 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:32,038 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. started... 2023-03-26 13:54:32,039 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-26 13:54:32,039 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:32,039 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:32,039 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:32,039 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:32,039 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:32,039 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:32,247 DEBUG [Thread-1042] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x204b5880 to 127.0.0.1:54332 2023-03-26 13:54:32,247 DEBUG [Thread-1042] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:32,249 DEBUG [Thread-1038] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2668d41d to 127.0.0.1:54332 2023-03-26 13:54:32,249 DEBUG [Thread-1038] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:32,250 DEBUG [Thread-1040] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b416c44 to 127.0.0.1:54332 2023-03-26 13:54:32,250 DEBUG [Thread-1040] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:32,251 DEBUG [Thread-1044] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41173aec to 127.0.0.1:54332 2023-03-26 13:54:32,251 DEBUG [Thread-1044] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:32,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:32,452 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. as already flushing 2023-03-26 13:54:32,452 DEBUG [Thread-1033] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25441895 to 127.0.0.1:54332 2023-03-26 13:54:32,453 DEBUG [Thread-1033] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:32,457 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326228f27d637cd43b782ab141b7ae73dbe to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326228f27d637cd43b782ab141b7ae73dbe 2023-03-26 13:54:32,461 DEBUG [Thread-1035] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40e0a3dc to 127.0.0.1:54332 2023-03-26 13:54:32,461 DEBUG [Thread-1035] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:32,462 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=581, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/901d543afc5142a8b317fb67eec7a22f 2023-03-26 13:54:32,472 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/c231b46aa859492f8dfdb5e8d7ff942a 2023-03-26 13:54:32,882 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/4b91460f646143d296f6ddd294f4c020 2023-03-26 13:54:32,886 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/901d543afc5142a8b317fb67eec7a22f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/901d543afc5142a8b317fb67eec7a22f 2023-03-26 13:54:32,890 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/901d543afc5142a8b317fb67eec7a22f, entries=150, sequenceid=581, filesize=25.4 K 2023-03-26 13:54:32,891 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/c231b46aa859492f8dfdb5e8d7ff942a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/c231b46aa859492f8dfdb5e8d7ff942a 2023-03-26 13:54:32,894 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/c231b46aa859492f8dfdb5e8d7ff942a, entries=150, sequenceid=581, filesize=11.9 K 2023-03-26 13:54:32,895 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/4b91460f646143d296f6ddd294f4c020 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/4b91460f646143d296f6ddd294f4c020 2023-03-26 13:54:32,898 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/4b91460f646143d296f6ddd294f4c020, entries=150, sequenceid=581, filesize=11.9 K 2023-03-26 13:54:32,898 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=13.42 KB/13740 for a739eafaf8b68b3135ee013236515a80 in 860ms, sequenceid=581, compaction requested=true 2023-03-26 13:54:32,899 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:32,899 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:32,899 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:32,899 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:32,899 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:32,899 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:32,899 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:32,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:32,900 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:32,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:32,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:32,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:32,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:32,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:32,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:32,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:32,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:32,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:32,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:32,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:32,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11e1e09c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:32,902 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:32,902 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:32,903 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:32,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:32,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:32,903 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:32,903 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:32,903 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:32,903 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:32,903 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:32,905 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:32,905 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:32,905 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:32,905 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:32,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:32,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:32,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:32,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:32,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:32,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:32,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:32,907 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:32,907 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:32,907 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:32,907 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:32,907 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:32,911 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:32,911 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:32,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:32,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:32,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:32,911 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:32,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:32,912 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:32,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:32,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:32,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:32,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:32,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:32,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:32,913 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:32,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:32,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:32,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:32,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:32,915 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:32,915 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:32,915 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:32,915 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:32,915 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:32,915 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:32,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:32,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:32,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:32,915 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:32,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:32,915 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:32,915 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:32,915 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:32,915 DEBUG [Thread-1037] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:32,915 DEBUG [Thread-1037] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:32,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:32,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:33,015 DEBUG [Thread-1037] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:33,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:33,896 DEBUG [Thread-1029] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52d98fed to 127.0.0.1:54332 2023-03-26 13:54:33,896 DEBUG [Thread-1029] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:33,897 DEBUG [Thread-1027] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63ebf1e9 to 127.0.0.1:54332 2023-03-26 13:54:33,897 DEBUG [Thread-1027] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:33,899 DEBUG [Thread-1031] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18496e17 to 127.0.0.1:54332 2023-03-26 13:54:33,899 DEBUG [Thread-1031] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:33,899 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-26 13:54:33,899 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 101 2023-03-26 13:54:33,899 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 96 2023-03-26 13:54:33,899 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 85 2023-03-26 13:54:33,899 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 108 2023-03-26 13:54:33,900 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 81 2023-03-26 13:54:33,900 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-26 13:54:33,900 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(398): read 4734 2023-03-26 13:54:33,900 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(398): read 4685 2023-03-26 13:54:33,900 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-26 13:54:33,900 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(402): scanned 2041 2023-03-26 13:54:33,900 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(403): verified 6120 rows 2023-03-26 13:54:33,900 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(402): scanned 2014 2023-03-26 13:54:33,900 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(403): verified 6039 rows 2023-03-26 13:54:33,900 INFO [Listener at localhost/41531] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-26 13:54:33,900 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ab850e7 to 127.0.0.1:54332 2023-03-26 13:54:33,900 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:33,903 INFO [Listener at localhost/41531] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-26 13:54:33,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-26 13:54:33,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-26 13:54:33,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-26 13:54:33,908 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838873907"}]},"ts":"1679838873907"} 2023-03-26 13:54:33,909 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-26 13:54:33,910 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-26 13:54:33,911 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a739eafaf8b68b3135ee013236515a80, UNASSIGN}] 2023-03-26 13:54:33,912 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a739eafaf8b68b3135ee013236515a80, UNASSIGN 2023-03-26 13:54:33,912 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=a739eafaf8b68b3135ee013236515a80, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:33,912 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838873912"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838873912"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838873912"}]},"ts":"1679838873912"} 2023-03-26 13:54:33,913 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:54:34,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-26 13:54:34,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:34,065 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a739eafaf8b68b3135ee013236515a80, disabling compactions & flushes 2023-03-26 13:54:34,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:34,065 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:34,065 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. after waiting 0 ms 2023-03-26 13:54:34,065 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:34,065 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing a739eafaf8b68b3135ee013236515a80 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-26 13:54:34,065 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=A 2023-03-26 13:54:34,065 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:34,066 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=B 2023-03-26 13:54:34,066 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:34,066 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a739eafaf8b68b3135ee013236515a80, store=C 2023-03-26 13:54:34,066 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:34,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-26 13:54:34,476 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326ac15f313db77470abdd2226fef26c2b0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ac15f313db77470abdd2226fef26c2b0 2023-03-26 13:54:34,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-26 13:54:34,881 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=589, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/ae469420ca6d4bb6b05fded8c6f17d73 2023-03-26 13:54:35,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-26 13:54:35,290 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=589 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/a51e8df330f34bb59ddd4a71275ee413 2023-03-26 13:54:35,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=589 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/5a72316fbb8f428189d6a2f997dfa0da 2023-03-26 13:54:35,703 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/A/ae469420ca6d4bb6b05fded8c6f17d73 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/ae469420ca6d4bb6b05fded8c6f17d73 2023-03-26 13:54:35,705 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/ae469420ca6d4bb6b05fded8c6f17d73, entries=150, sequenceid=589, filesize=25.4 K 2023-03-26 13:54:35,706 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/B/a51e8df330f34bb59ddd4a71275ee413 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a51e8df330f34bb59ddd4a71275ee413 2023-03-26 13:54:35,709 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a51e8df330f34bb59ddd4a71275ee413, entries=150, sequenceid=589, filesize=11.9 K 2023-03-26 13:54:35,710 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/.tmp/C/5a72316fbb8f428189d6a2f997dfa0da as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/5a72316fbb8f428189d6a2f997dfa0da 2023-03-26 13:54:35,713 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/5a72316fbb8f428189d6a2f997dfa0da, entries=150, sequenceid=589, filesize=11.9 K 2023-03-26 13:54:35,714 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for a739eafaf8b68b3135ee013236515a80 in 1649ms, sequenceid=589, compaction requested=true 2023-03-26 13:54:35,735 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/63dd65ae30f64f0d8ea160ea9b8f55b9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/34016a20526c400d9cdff5c0ced61960, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/2fa5e34796e848a68253b58566e1c262, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/fd77731c5f924e6989a2daa22fcb96ac, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/9b65cc6695a245e9908bf41d67d6eb48, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/20acdfc0348a41e5bb1345f88ce26633, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/519a5a9e89134998affe1ad0fbd0c77f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/ab79e1d726ad4b4f936db947a46d6c71, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/44e1d917e3ee4319a1a5b813e81ccbd9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/33b03d23e9f84ca8990d4f865acab751, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/511d71e8beab47e2972c60327aa36174, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/f63e6a8e0b1148f9936084c736ba6653, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/9844b4f839324f459d3d4131f82db3fe, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/06f6eaa81b5b44f6a80e81753fd0a964, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/b67efe2cff6244faa1780a08dee1517c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/ba410f580e7d495e8b38d565b6703465, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/2720d727e11f4be09854792776fe8d03, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/145f162063df4e89bca44bb6c741e7f5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/7796cf7ae41244e58cb3781708073c42, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/a71a5ccb08f04594bb58e655e8dea68c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/fee5081a5dd6417a89378994cbbb62d3, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/0fd097c52e1b463b8a52149f98d3c3ff, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/493884f3c0594b47a02dc687d4c0d2ff, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/32b9b27cc7774158b3045f1947134c4e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/6161d677b6bc43439eea5778a3cce27c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/be7d1790fe474393a2180ff2e4080807, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/87cc6fe787284eb288853e0ecc17241a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/5099275104444f77b3dbef38fddf9952, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/87a5e52f570f493da3c8b59736f12661, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/5a31f877a04746bb8fd630099e7acd79] to archive 2023-03-26 13:54:35,736 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 13:54:35,738 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/63dd65ae30f64f0d8ea160ea9b8f55b9 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/63dd65ae30f64f0d8ea160ea9b8f55b9 2023-03-26 13:54:35,739 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/34016a20526c400d9cdff5c0ced61960 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/34016a20526c400d9cdff5c0ced61960 2023-03-26 13:54:35,740 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/2fa5e34796e848a68253b58566e1c262 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/2fa5e34796e848a68253b58566e1c262 2023-03-26 13:54:35,741 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/fd77731c5f924e6989a2daa22fcb96ac to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/fd77731c5f924e6989a2daa22fcb96ac 2023-03-26 13:54:35,741 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/9b65cc6695a245e9908bf41d67d6eb48 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/9b65cc6695a245e9908bf41d67d6eb48 2023-03-26 13:54:35,742 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/20acdfc0348a41e5bb1345f88ce26633 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/20acdfc0348a41e5bb1345f88ce26633 2023-03-26 13:54:35,743 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/519a5a9e89134998affe1ad0fbd0c77f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/519a5a9e89134998affe1ad0fbd0c77f 2023-03-26 13:54:35,744 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/ab79e1d726ad4b4f936db947a46d6c71 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/ab79e1d726ad4b4f936db947a46d6c71 2023-03-26 13:54:35,745 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/44e1d917e3ee4319a1a5b813e81ccbd9 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/44e1d917e3ee4319a1a5b813e81ccbd9 2023-03-26 13:54:35,746 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/33b03d23e9f84ca8990d4f865acab751 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/33b03d23e9f84ca8990d4f865acab751 2023-03-26 13:54:35,747 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/511d71e8beab47e2972c60327aa36174 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/511d71e8beab47e2972c60327aa36174 2023-03-26 13:54:35,748 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/f63e6a8e0b1148f9936084c736ba6653 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/f63e6a8e0b1148f9936084c736ba6653 2023-03-26 13:54:35,749 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/9844b4f839324f459d3d4131f82db3fe to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/9844b4f839324f459d3d4131f82db3fe 2023-03-26 13:54:35,750 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/06f6eaa81b5b44f6a80e81753fd0a964 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/06f6eaa81b5b44f6a80e81753fd0a964 2023-03-26 13:54:35,751 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/b67efe2cff6244faa1780a08dee1517c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/b67efe2cff6244faa1780a08dee1517c 2023-03-26 13:54:35,752 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/ba410f580e7d495e8b38d565b6703465 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/ba410f580e7d495e8b38d565b6703465 2023-03-26 13:54:35,753 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/2720d727e11f4be09854792776fe8d03 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/2720d727e11f4be09854792776fe8d03 2023-03-26 13:54:35,754 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/145f162063df4e89bca44bb6c741e7f5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/145f162063df4e89bca44bb6c741e7f5 2023-03-26 13:54:35,755 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/7796cf7ae41244e58cb3781708073c42 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/7796cf7ae41244e58cb3781708073c42 2023-03-26 13:54:35,756 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/a71a5ccb08f04594bb58e655e8dea68c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/a71a5ccb08f04594bb58e655e8dea68c 2023-03-26 13:54:35,757 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/fee5081a5dd6417a89378994cbbb62d3 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/fee5081a5dd6417a89378994cbbb62d3 2023-03-26 13:54:35,758 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/0fd097c52e1b463b8a52149f98d3c3ff to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/0fd097c52e1b463b8a52149f98d3c3ff 2023-03-26 13:54:35,759 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/493884f3c0594b47a02dc687d4c0d2ff to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/493884f3c0594b47a02dc687d4c0d2ff 2023-03-26 13:54:35,760 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/32b9b27cc7774158b3045f1947134c4e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/32b9b27cc7774158b3045f1947134c4e 2023-03-26 13:54:35,761 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/6161d677b6bc43439eea5778a3cce27c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/6161d677b6bc43439eea5778a3cce27c 2023-03-26 13:54:35,762 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/be7d1790fe474393a2180ff2e4080807 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/be7d1790fe474393a2180ff2e4080807 2023-03-26 13:54:35,762 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/87cc6fe787284eb288853e0ecc17241a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/87cc6fe787284eb288853e0ecc17241a 2023-03-26 13:54:35,763 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/5099275104444f77b3dbef38fddf9952 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/5099275104444f77b3dbef38fddf9952 2023-03-26 13:54:35,764 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/87a5e52f570f493da3c8b59736f12661 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/87a5e52f570f493da3c8b59736f12661 2023-03-26 13:54:35,765 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/5a31f877a04746bb8fd630099e7acd79 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/5a31f877a04746bb8fd630099e7acd79 2023-03-26 13:54:35,768 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/4fb2e4ac1db9405c8f93f262744b933c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/13dd805b5c7347e0ab11e3ab5da9fc9c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/8472c33a12c7483eb3f4f5d8efbd632e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/e73de165ec8f475da06efc9533e962e4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/990c9016eee24195b526aa1debefcf0f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/767ed78b783d41989d740fd1e8781969, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/06dfd74305b34ec8aa19275ecfe7bae6, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/f95f75f560da416a8a692087006d7ae0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/89ab3dee95ce4b65a14a1d6b6db35edd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/44b1fab7bb964146881fcc5c9c40bd12, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a0027c460d73498f8ff22c201e2c0097, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/3a835a4081a94491b51b91e627bee0bc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/96127f7d438a474794cb3d6b65892a7c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/7f14380d83334d16aa09af2979b1108a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/78a693a12d7b466a96a7ddb162878939, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a38f4a123b424eae9f58a37eed4c109c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/779bf18501894f5985d67ace90fe8bb0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/386d137d738542618e5ebfed757401fd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/2dc71f187f374347b17400753ae2a8f9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/193baf37023d459bb6bf637683b6469c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/c1cec8fed9ba49f784d264d5b2db7690, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/791aa25ad0084358a730ea359ac83beb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/63db185ceb6441748a00f5bb6156b59e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/01ab8f9b420a4b8c9380a4d1a0ebb505, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/25f5a355008b4349a36d2b901b1a011e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a3f20253f34b446b89665ee0f6a36b70, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/4e4e3861313047f48709e75898e3d15e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/bcffdad9d36c4d6cbbcb6f8456a90325, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/75a20c01eab44cf1a8a0685f32206a9b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/bce56b702c074788819b2e60e72df575] to archive 2023-03-26 13:54:35,769 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 13:54:35,770 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/4fb2e4ac1db9405c8f93f262744b933c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/4fb2e4ac1db9405c8f93f262744b933c 2023-03-26 13:54:35,771 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/13dd805b5c7347e0ab11e3ab5da9fc9c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/13dd805b5c7347e0ab11e3ab5da9fc9c 2023-03-26 13:54:35,772 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/8472c33a12c7483eb3f4f5d8efbd632e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/8472c33a12c7483eb3f4f5d8efbd632e 2023-03-26 13:54:35,773 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/e73de165ec8f475da06efc9533e962e4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/e73de165ec8f475da06efc9533e962e4 2023-03-26 13:54:35,774 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/990c9016eee24195b526aa1debefcf0f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/990c9016eee24195b526aa1debefcf0f 2023-03-26 13:54:35,774 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/767ed78b783d41989d740fd1e8781969 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/767ed78b783d41989d740fd1e8781969 2023-03-26 13:54:35,775 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/06dfd74305b34ec8aa19275ecfe7bae6 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/06dfd74305b34ec8aa19275ecfe7bae6 2023-03-26 13:54:35,776 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/f95f75f560da416a8a692087006d7ae0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/f95f75f560da416a8a692087006d7ae0 2023-03-26 13:54:35,777 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/89ab3dee95ce4b65a14a1d6b6db35edd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/89ab3dee95ce4b65a14a1d6b6db35edd 2023-03-26 13:54:35,778 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/44b1fab7bb964146881fcc5c9c40bd12 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/44b1fab7bb964146881fcc5c9c40bd12 2023-03-26 13:54:35,779 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a0027c460d73498f8ff22c201e2c0097 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a0027c460d73498f8ff22c201e2c0097 2023-03-26 13:54:35,780 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/3a835a4081a94491b51b91e627bee0bc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/3a835a4081a94491b51b91e627bee0bc 2023-03-26 13:54:35,781 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/96127f7d438a474794cb3d6b65892a7c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/96127f7d438a474794cb3d6b65892a7c 2023-03-26 13:54:35,782 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/7f14380d83334d16aa09af2979b1108a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/7f14380d83334d16aa09af2979b1108a 2023-03-26 13:54:35,783 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/78a693a12d7b466a96a7ddb162878939 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/78a693a12d7b466a96a7ddb162878939 2023-03-26 13:54:35,784 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a38f4a123b424eae9f58a37eed4c109c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a38f4a123b424eae9f58a37eed4c109c 2023-03-26 13:54:35,796 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/779bf18501894f5985d67ace90fe8bb0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/779bf18501894f5985d67ace90fe8bb0 2023-03-26 13:54:35,799 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/386d137d738542618e5ebfed757401fd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/386d137d738542618e5ebfed757401fd 2023-03-26 13:54:35,800 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/2dc71f187f374347b17400753ae2a8f9 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/2dc71f187f374347b17400753ae2a8f9 2023-03-26 13:54:35,802 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/193baf37023d459bb6bf637683b6469c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/193baf37023d459bb6bf637683b6469c 2023-03-26 13:54:35,803 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/c1cec8fed9ba49f784d264d5b2db7690 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/c1cec8fed9ba49f784d264d5b2db7690 2023-03-26 13:54:35,804 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/791aa25ad0084358a730ea359ac83beb to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/791aa25ad0084358a730ea359ac83beb 2023-03-26 13:54:35,805 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/63db185ceb6441748a00f5bb6156b59e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/63db185ceb6441748a00f5bb6156b59e 2023-03-26 13:54:35,806 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/01ab8f9b420a4b8c9380a4d1a0ebb505 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/01ab8f9b420a4b8c9380a4d1a0ebb505 2023-03-26 13:54:35,807 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/25f5a355008b4349a36d2b901b1a011e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/25f5a355008b4349a36d2b901b1a011e 2023-03-26 13:54:35,808 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a3f20253f34b446b89665ee0f6a36b70 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a3f20253f34b446b89665ee0f6a36b70 2023-03-26 13:54:35,809 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/4e4e3861313047f48709e75898e3d15e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/4e4e3861313047f48709e75898e3d15e 2023-03-26 13:54:35,810 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/bcffdad9d36c4d6cbbcb6f8456a90325 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/bcffdad9d36c4d6cbbcb6f8456a90325 2023-03-26 13:54:35,812 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/75a20c01eab44cf1a8a0685f32206a9b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/75a20c01eab44cf1a8a0685f32206a9b 2023-03-26 13:54:35,813 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/bce56b702c074788819b2e60e72df575 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/bce56b702c074788819b2e60e72df575 2023-03-26 13:54:35,816 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/22e30b94be774f1092dbf9f2255e4cbb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/55c6fb3d81f241d3abcebc3ed7254fee, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/172f8a97f0c54fdd809e3271c470dd80, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/63368c71628741af83a8929e0528f8b3, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/06274ad312e34c2f85d1dfd9a144b15b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/4cfc9bb3ec624560a4dd36addacd78fc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/a0ed65be43e446768509dbf67f85237a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/8bd94a0333754397817cc8b378ea3c56, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/a73bcb7d8c2742788f97bf9b8d310e7f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/c29f7f7247e348aba502696491bc1e67, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/f4cf4bfc80b44a16af435661fcf060b9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/b468f7e0253f4c33b5936392568990af, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/f5d5b77dc9fe4693b7e834e736d83083, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/e12d3c8da70a450687d0089e63df6363, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/31073bd57dc549eb9ed4548bf11584cb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/8811323f7fdc468081f15e60f6d34d5d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/b847397582af4a1db256af03466e7989, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/2de3ca8d707e488a95b99b2d0a0a58d4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/56a5332896ed4e70ab6c0f53f18ed9f8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/b5203f851f80423ca53e3ce37bb7465c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/e636bcbb84f84556b105ca1f97e883cc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/96984235d85942f59bd2adb50beb705e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/5143513770fe4afcbe5bfc26adc9dd71, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/d72b3327c14e4809a75731a56cb1138f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/30b2dfe9e28a477cb4090a534df0a0e0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/798cff5240c346f3a353c24ca71db73f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/5a2e190e508b4d02a56a4574103cf7b2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/2483557bf34e433cbb388bd8ca10fd1b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/64de000ed33f4405b6a52ab30d903f15, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/7c02a820de684cd99bad0351431783c9] to archive 2023-03-26 13:54:35,817 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 13:54:35,819 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/22e30b94be774f1092dbf9f2255e4cbb to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/22e30b94be774f1092dbf9f2255e4cbb 2023-03-26 13:54:35,820 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/55c6fb3d81f241d3abcebc3ed7254fee to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/55c6fb3d81f241d3abcebc3ed7254fee 2023-03-26 13:54:35,821 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/172f8a97f0c54fdd809e3271c470dd80 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/172f8a97f0c54fdd809e3271c470dd80 2023-03-26 13:54:35,823 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/63368c71628741af83a8929e0528f8b3 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/63368c71628741af83a8929e0528f8b3 2023-03-26 13:54:35,824 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/06274ad312e34c2f85d1dfd9a144b15b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/06274ad312e34c2f85d1dfd9a144b15b 2023-03-26 13:54:35,825 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/4cfc9bb3ec624560a4dd36addacd78fc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/4cfc9bb3ec624560a4dd36addacd78fc 2023-03-26 13:54:35,826 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/a0ed65be43e446768509dbf67f85237a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/a0ed65be43e446768509dbf67f85237a 2023-03-26 13:54:35,827 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/8bd94a0333754397817cc8b378ea3c56 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/8bd94a0333754397817cc8b378ea3c56 2023-03-26 13:54:35,828 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/a73bcb7d8c2742788f97bf9b8d310e7f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/a73bcb7d8c2742788f97bf9b8d310e7f 2023-03-26 13:54:35,829 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/c29f7f7247e348aba502696491bc1e67 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/c29f7f7247e348aba502696491bc1e67 2023-03-26 13:54:35,830 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/f4cf4bfc80b44a16af435661fcf060b9 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/f4cf4bfc80b44a16af435661fcf060b9 2023-03-26 13:54:35,831 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/b468f7e0253f4c33b5936392568990af to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/b468f7e0253f4c33b5936392568990af 2023-03-26 13:54:35,832 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/f5d5b77dc9fe4693b7e834e736d83083 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/f5d5b77dc9fe4693b7e834e736d83083 2023-03-26 13:54:35,833 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/e12d3c8da70a450687d0089e63df6363 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/e12d3c8da70a450687d0089e63df6363 2023-03-26 13:54:35,834 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/31073bd57dc549eb9ed4548bf11584cb to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/31073bd57dc549eb9ed4548bf11584cb 2023-03-26 13:54:35,835 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/8811323f7fdc468081f15e60f6d34d5d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/8811323f7fdc468081f15e60f6d34d5d 2023-03-26 13:54:35,836 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/b847397582af4a1db256af03466e7989 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/b847397582af4a1db256af03466e7989 2023-03-26 13:54:35,837 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/2de3ca8d707e488a95b99b2d0a0a58d4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/2de3ca8d707e488a95b99b2d0a0a58d4 2023-03-26 13:54:35,838 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/56a5332896ed4e70ab6c0f53f18ed9f8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/56a5332896ed4e70ab6c0f53f18ed9f8 2023-03-26 13:54:35,839 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/b5203f851f80423ca53e3ce37bb7465c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/b5203f851f80423ca53e3ce37bb7465c 2023-03-26 13:54:35,841 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/e636bcbb84f84556b105ca1f97e883cc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/e636bcbb84f84556b105ca1f97e883cc 2023-03-26 13:54:35,842 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/96984235d85942f59bd2adb50beb705e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/96984235d85942f59bd2adb50beb705e 2023-03-26 13:54:35,843 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/5143513770fe4afcbe5bfc26adc9dd71 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/5143513770fe4afcbe5bfc26adc9dd71 2023-03-26 13:54:35,844 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/d72b3327c14e4809a75731a56cb1138f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/d72b3327c14e4809a75731a56cb1138f 2023-03-26 13:54:35,845 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/30b2dfe9e28a477cb4090a534df0a0e0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/30b2dfe9e28a477cb4090a534df0a0e0 2023-03-26 13:54:35,846 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/798cff5240c346f3a353c24ca71db73f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/798cff5240c346f3a353c24ca71db73f 2023-03-26 13:54:35,847 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/5a2e190e508b4d02a56a4574103cf7b2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/5a2e190e508b4d02a56a4574103cf7b2 2023-03-26 13:54:35,849 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/2483557bf34e433cbb388bd8ca10fd1b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/2483557bf34e433cbb388bd8ca10fd1b 2023-03-26 13:54:35,850 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/64de000ed33f4405b6a52ab30d903f15 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/64de000ed33f4405b6a52ab30d903f15 2023-03-26 13:54:35,851 DEBUG [StoreCloser-TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/7c02a820de684cd99bad0351431783c9 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/7c02a820de684cd99bad0351431783c9 2023-03-26 13:54:35,855 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/recovered.edits/592.seqid, newMaxSeqId=592, maxSeqId=4 2023-03-26 13:54:35,857 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80. 2023-03-26 13:54:35,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a739eafaf8b68b3135ee013236515a80: 2023-03-26 13:54:35,858 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:35,858 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=a739eafaf8b68b3135ee013236515a80, regionState=CLOSED 2023-03-26 13:54:35,859 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838875858"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679838875858"}]},"ts":"1679838875858"} 2023-03-26 13:54:35,861 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-03-26 13:54:35,861 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure a739eafaf8b68b3135ee013236515a80, server=jenkins-hbase4.apache.org,42517,1679838814721 in 1.9460 sec 2023-03-26 13:54:35,862 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-03-26 13:54:35,862 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a739eafaf8b68b3135ee013236515a80, UNASSIGN in 1.9500 sec 2023-03-26 13:54:35,862 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838875862"}]},"ts":"1679838875862"} 2023-03-26 13:54:35,863 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-26 13:54:35,865 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-26 13:54:35,866 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9610 sec 2023-03-26 13:54:36,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-26 13:54:36,012 INFO [Listener at localhost/41531] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-03-26 13:54:36,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-26 13:54:36,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:54:36,014 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:54:36,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-26 13:54:36,015 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:54:36,020 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:36,022 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A, FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B, FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C, FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/recovered.edits] 2023-03-26 13:54:36,024 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/821b5ffd1a05423483741a0df84ba9ea to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/821b5ffd1a05423483741a0df84ba9ea 2023-03-26 13:54:36,026 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/901d543afc5142a8b317fb67eec7a22f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/901d543afc5142a8b317fb67eec7a22f 2023-03-26 13:54:36,027 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/90646c01d4a544188595d8d69a9528e4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/90646c01d4a544188595d8d69a9528e4 2023-03-26 13:54:36,028 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/adab2b725fee4664aa8e2b8bc86abea8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/adab2b725fee4664aa8e2b8bc86abea8 2023-03-26 13:54:36,029 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/ae469420ca6d4bb6b05fded8c6f17d73 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/A/ae469420ca6d4bb6b05fded8c6f17d73 2023-03-26 13:54:36,031 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/0fc99758c7d6444abf0575b74fc61135 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/0fc99758c7d6444abf0575b74fc61135 2023-03-26 13:54:36,032 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/2e74822727b64c878a0bc99d4914a575 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/2e74822727b64c878a0bc99d4914a575 2023-03-26 13:54:36,033 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/49f3f0bc2b25418198a624d7cc9b397b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/49f3f0bc2b25418198a624d7cc9b397b 2023-03-26 13:54:36,035 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a51e8df330f34bb59ddd4a71275ee413 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/a51e8df330f34bb59ddd4a71275ee413 2023-03-26 13:54:36,036 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/c231b46aa859492f8dfdb5e8d7ff942a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/B/c231b46aa859492f8dfdb5e8d7ff942a 2023-03-26 13:54:36,036 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-26 13:54:36,038 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/2b1b7dd7bd10434d831e3f2385703975 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/2b1b7dd7bd10434d831e3f2385703975 2023-03-26 13:54:36,039 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/4a7677d885904fe1ba97aadb31012867 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/4a7677d885904fe1ba97aadb31012867 2023-03-26 13:54:36,041 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/4b91460f646143d296f6ddd294f4c020 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/4b91460f646143d296f6ddd294f4c020 2023-03-26 13:54:36,042 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/5a72316fbb8f428189d6a2f997dfa0da to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/5a72316fbb8f428189d6a2f997dfa0da 2023-03-26 13:54:36,044 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/a3deccec3dc84a1293a635fc98dafa7a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/C/a3deccec3dc84a1293a635fc98dafa7a 2023-03-26 13:54:36,047 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/recovered.edits/592.seqid to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80/recovered.edits/592.seqid 2023-03-26 13:54:36,048 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a739eafaf8b68b3135ee013236515a80 2023-03-26 13:54:36,048 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 13:54:36,049 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-26 13:54:36,050 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-26 13:54:36,054 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032603f36650b3764a04a5afa99a5f4011f5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032603f36650b3764a04a5afa99a5f4011f5 2023-03-26 13:54:36,055 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260c541cd74cd34977b17fa6a94994f19a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260c541cd74cd34977b17fa6a94994f19a 2023-03-26 13:54:36,057 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303262037996c9ff4438989c67f74ef383b91 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303262037996c9ff4438989c67f74ef383b91 2023-03-26 13:54:36,058 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326228f27d637cd43b782ab141b7ae73dbe to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326228f27d637cd43b782ab141b7ae73dbe 2023-03-26 13:54:36,060 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032637f853ebb44c48a58a8750ff8fbae3f6 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032637f853ebb44c48a58a8750ff8fbae3f6 2023-03-26 13:54:36,061 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326383022ad871343e193b19c1e5fc84297 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326383022ad871343e193b19c1e5fc84297 2023-03-26 13:54:36,062 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263a0e15d290404db69d56b9cf5e7c802a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263a0e15d290404db69d56b9cf5e7c802a 2023-03-26 13:54:36,064 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032656f1d3a697384a2e95df940b4232c573 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032656f1d3a697384a2e95df940b4232c573 2023-03-26 13:54:36,065 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326687167ef77854fd3a0cc568e62632436 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326687167ef77854fd3a0cc568e62632436 2023-03-26 13:54:36,067 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267c2eac9f3fbe4cf5bf707ed88c3d6ac0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267c2eac9f3fbe4cf5bf707ed88c3d6ac0 2023-03-26 13:54:36,068 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032681302e36bec44698acf7dfbc6ad88458 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032681302e36bec44698acf7dfbc6ad88458 2023-03-26 13:54:36,069 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032686295bb267b345f9a201fae295de1c93 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032686295bb267b345f9a201fae295de1c93 2023-03-26 13:54:36,071 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303268daf3607f1c44b8caff7b21150665c8f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303268daf3607f1c44b8caff7b21150665c8f 2023-03-26 13:54:36,077 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032693b23e7fc1774e1482060b37fc8f352e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032693b23e7fc1774e1482060b37fc8f352e 2023-03-26 13:54:36,078 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a1bd9b3c52aa430ba06c6bcdd9b0fae4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a1bd9b3c52aa430ba06c6bcdd9b0fae4 2023-03-26 13:54:36,079 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ab2c5e4889a94117a7d3a4f6dd05cf5f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ab2c5e4889a94117a7d3a4f6dd05cf5f 2023-03-26 13:54:36,081 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ac15f313db77470abdd2226fef26c2b0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ac15f313db77470abdd2226fef26c2b0 2023-03-26 13:54:36,082 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ad58506847bf476baa7b2ac8203dfe04 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ad58506847bf476baa7b2ac8203dfe04 2023-03-26 13:54:36,083 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326af3b4b60a49648ecbbc03ed2f0c4de12 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326af3b4b60a49648ecbbc03ed2f0c4de12 2023-03-26 13:54:36,084 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326bf02f9b5488344b29cc7863f122f2ce2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326bf02f9b5488344b29cc7863f122f2ce2 2023-03-26 13:54:36,085 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326c47f93f178bc49f0af7a835d315702db to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326c47f93f178bc49f0af7a835d315702db 2023-03-26 13:54:36,086 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326c960814eee4644e890812d99466736b3 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326c960814eee4644e890812d99466736b3 2023-03-26 13:54:36,087 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326d1fff24f1d294f6a9428105c2cb8af80 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326d1fff24f1d294f6a9428105c2cb8af80 2023-03-26 13:54:36,088 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e39abc346fd64fdb9adc39bc8e4997d7 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e39abc346fd64fdb9adc39bc8e4997d7 2023-03-26 13:54:36,089 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ed338cb8a96f471d88d032976bad7feb to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ed338cb8a96f471d88d032976bad7feb 2023-03-26 13:54:36,090 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f2ce51ccc13b47aa9079653cef9f976d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f2ce51ccc13b47aa9079653cef9f976d 2023-03-26 13:54:36,091 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f2e56ee592b74ad09394747c3a1d8712 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f2e56ee592b74ad09394747c3a1d8712 2023-03-26 13:54:36,092 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-26 13:54:36,094 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:54:36,096 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-26 13:54:36,098 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-26 13:54:36,099 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:54:36,099 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-26 13:54:36,099 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679838876099"}]},"ts":"9223372036854775807"} 2023-03-26 13:54:36,100 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-26 13:54:36,100 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a739eafaf8b68b3135ee013236515a80, NAME => 'TestAcidGuarantees,,1679838849117.a739eafaf8b68b3135ee013236515a80.', STARTKEY => '', ENDKEY => ''}] 2023-03-26 13:54:36,100 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-26 13:54:36,100 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679838876100"}]},"ts":"9223372036854775807"} 2023-03-26 13:54:36,102 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-26 13:54:36,104 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:54:36,105 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 92 msec 2023-03-26 13:54:36,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-26 13:54:36,116 INFO [Listener at localhost/41531] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-03-26 13:54:36,130 INFO [Listener at localhost/41531] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=232 (was 231) - Thread LEAK? -, OpenFileDescriptor=591 (was 545) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=472 (was 434) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=8821 (was 9096) 2023-03-26 13:54:36,141 INFO [Listener at localhost/41531] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=232, OpenFileDescriptor=591, MaxFileDescriptor=60000, SystemLoadAverage=472, ProcessCount=172, AvailableMemoryMB=8821 2023-03-26 13:54:36,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-26 13:54:36,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 13:54:36,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-26 13:54:36,146 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-26 13:54:36,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-03-26 13:54:36,146 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-26 13:54:36,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-26 13:54:36,147 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:36,148 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d empty. 2023-03-26 13:54:36,148 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:36,148 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 13:54:36,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-26 13:54:36,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-26 13:54:36,556 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-26 13:54:36,558 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a59be4541fa5742a2bd937206749175d, NAME => 'TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp 2023-03-26 13:54:36,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-26 13:54:36,963 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:54:36,963 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a59be4541fa5742a2bd937206749175d, disabling compactions & flushes 2023-03-26 13:54:36,963 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:36,963 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:36,963 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. after waiting 0 ms 2023-03-26 13:54:36,963 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:36,963 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:36,963 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:36,965 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-26 13:54:36,966 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838876966"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679838876966"}]},"ts":"1679838876966"} 2023-03-26 13:54:36,967 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-26 13:54:36,968 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-26 13:54:36,968 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838876968"}]},"ts":"1679838876968"} 2023-03-26 13:54:36,969 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-26 13:54:36,971 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a59be4541fa5742a2bd937206749175d, ASSIGN}] 2023-03-26 13:54:36,972 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a59be4541fa5742a2bd937206749175d, ASSIGN 2023-03-26 13:54:36,973 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a59be4541fa5742a2bd937206749175d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42517,1679838814721; forceNewPlan=false, retain=false 2023-03-26 13:54:37,124 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=a59be4541fa5742a2bd937206749175d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:37,124 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838877124"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838877124"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838877124"}]},"ts":"1679838877124"} 2023-03-26 13:54:37,126 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:54:37,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-26 13:54:37,281 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:37,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a59be4541fa5742a2bd937206749175d, NAME => 'TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.', STARTKEY => '', ENDKEY => ''} 2023-03-26 13:54:37,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:37,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:54:37,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:37,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:37,283 INFO [StoreOpener-a59be4541fa5742a2bd937206749175d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:37,285 DEBUG [StoreOpener-a59be4541fa5742a2bd937206749175d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A 2023-03-26 13:54:37,285 DEBUG [StoreOpener-a59be4541fa5742a2bd937206749175d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A 2023-03-26 13:54:37,285 INFO [StoreOpener-a59be4541fa5742a2bd937206749175d-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:54:37,285 INFO [StoreOpener-a59be4541fa5742a2bd937206749175d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a59be4541fa5742a2bd937206749175d columnFamilyName A 2023-03-26 13:54:37,286 INFO [StoreOpener-a59be4541fa5742a2bd937206749175d-1] regionserver.HStore(310): Store=a59be4541fa5742a2bd937206749175d/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:54:37,286 INFO [StoreOpener-a59be4541fa5742a2bd937206749175d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:37,287 DEBUG [StoreOpener-a59be4541fa5742a2bd937206749175d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B 2023-03-26 13:54:37,287 DEBUG [StoreOpener-a59be4541fa5742a2bd937206749175d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B 2023-03-26 13:54:37,287 INFO [StoreOpener-a59be4541fa5742a2bd937206749175d-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:54:37,288 INFO [StoreOpener-a59be4541fa5742a2bd937206749175d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a59be4541fa5742a2bd937206749175d columnFamilyName B 2023-03-26 13:54:37,288 INFO [StoreOpener-a59be4541fa5742a2bd937206749175d-1] regionserver.HStore(310): Store=a59be4541fa5742a2bd937206749175d/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:54:37,288 INFO [StoreOpener-a59be4541fa5742a2bd937206749175d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:37,289 DEBUG [StoreOpener-a59be4541fa5742a2bd937206749175d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C 2023-03-26 13:54:37,289 DEBUG [StoreOpener-a59be4541fa5742a2bd937206749175d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C 2023-03-26 13:54:37,289 INFO [StoreOpener-a59be4541fa5742a2bd937206749175d-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:54:37,289 INFO [StoreOpener-a59be4541fa5742a2bd937206749175d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a59be4541fa5742a2bd937206749175d columnFamilyName C 2023-03-26 13:54:37,290 INFO [StoreOpener-a59be4541fa5742a2bd937206749175d-1] regionserver.HStore(310): Store=a59be4541fa5742a2bd937206749175d/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:54:37,290 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:37,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:37,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:37,292 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 13:54:37,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:37,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-26 13:54:37,295 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a59be4541fa5742a2bd937206749175d; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70000915, jitterRate=0.04309491813182831}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 13:54:37,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:37,296 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., pid=30, masterSystemTime=1679838877277 2023-03-26 13:54:37,297 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:37,297 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:37,297 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=a59be4541fa5742a2bd937206749175d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:37,297 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838877297"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679838877297"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679838877297"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679838877297"}]},"ts":"1679838877297"} 2023-03-26 13:54:37,299 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-03-26 13:54:37,299 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 in 172 msec 2023-03-26 13:54:37,301 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-26 13:54:37,301 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a59be4541fa5742a2bd937206749175d, ASSIGN in 328 msec 2023-03-26 13:54:37,301 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-26 13:54:37,301 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838877301"}]},"ts":"1679838877301"} 2023-03-26 13:54:37,302 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-26 13:54:37,305 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-26 13:54:37,306 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-03-26 13:54:38,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-26 13:54:38,251 INFO [Listener at localhost/41531] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-03-26 13:54:38,251 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x5b4a16d4 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:38,256 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13544701, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:38,257 DEBUG [Listener at localhost/41531] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:38,259 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34732, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:38,259 DEBUG [Listener at localhost/41531] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-26 13:54:38,261 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57324, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-26 13:54:38,262 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x1b40c526 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:38,266 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10b79c2c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:38,266 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x47ae15dc to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:38,271 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a5a81fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:38,271 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x025522dc to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:38,274 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3562c7bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:38,274 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x4c6d5986 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:38,278 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6705d115, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:38,278 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x31182447 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:38,281 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@179ffa8b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:38,282 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x270a3c0c to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:38,289 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4f29c1c4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:38,290 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x1e067bfc to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:38,294 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@681ace6c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:38,294 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x06bbe763 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:38,298 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2b4fa5d3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:38,299 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x1c8736d4 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:38,305 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45de5ad8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:38,306 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x451c882f to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:54:38,310 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@608837ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:54:38,314 DEBUG [hconnection-0x4e8f8ba3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:38,315 DEBUG [hconnection-0x7e657468-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:38,316 DEBUG [hconnection-0x36ce457a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:38,316 DEBUG [hconnection-0x4a7528b1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:38,316 DEBUG [hconnection-0x38dccb83-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:38,317 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34740, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:38,318 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34764, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:38,318 DEBUG [hconnection-0x6cdb9c56-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:38,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:38,318 DEBUG [hconnection-0x2a8d8dca-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:38,319 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34794, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:38,324 DEBUG [hconnection-0x25473759-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:38,326 DEBUG [hconnection-0x6480942-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:38,326 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34756, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:38,326 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34784, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:38,327 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34804, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:38,327 DEBUG [hconnection-0x56675b11-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:54:38,327 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34774, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:38,328 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34796, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:38,330 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34798, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:38,331 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34814, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:54:38,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:38,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:38,336 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:38,336 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:38,336 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:38,336 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:38,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,338 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:38,338 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,339 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,339 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:38,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:38,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:38,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:38,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:38,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:38,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:38,341 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:38,341 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:38,341 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:38,346 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:38,346 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:38,346 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:38,346 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:38,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:38,348 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 13:54:38,348 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:38,348 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:38,350 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:38,350 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:38,350 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:38,350 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:38,350 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:38,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:38,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838878356, max timestamp=1679838878367Num uniques 859; 2023-03-26 13:54:38,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838938365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:54:38,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838878356, max timestamp=1679838878368Num uniques 859; 2023-03-26 13:54:38,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:54:38,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838878356, max timestamp=1679838878368Num uniques 859; 2023-03-26 13:54:38,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:54:38,373 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/32c04a2eb6594ac29a6f943e6dcaed5b 2023-03-26 13:54:38,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:38,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838938386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:38,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838938389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,392 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:38,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838938390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:38,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838938391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,406 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/6008e0a6bea7449f8625c72625dc4c36 2023-03-26 13:54:38,426 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/0921b04106fd4d21a4d3497af4fea9cc 2023-03-26 13:54:38,432 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/32c04a2eb6594ac29a6f943e6dcaed5b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/32c04a2eb6594ac29a6f943e6dcaed5b 2023-03-26 13:54:38,437 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/32c04a2eb6594ac29a6f943e6dcaed5b, entries=150, sequenceid=12, filesize=11.6 K 2023-03-26 13:54:38,438 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/6008e0a6bea7449f8625c72625dc4c36 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6008e0a6bea7449f8625c72625dc4c36 2023-03-26 13:54:38,444 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6008e0a6bea7449f8625c72625dc4c36, entries=150, sequenceid=12, filesize=11.6 K 2023-03-26 13:54:38,445 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/0921b04106fd4d21a4d3497af4fea9cc as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/0921b04106fd4d21a4d3497af4fea9cc 2023-03-26 13:54:38,449 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/0921b04106fd4d21a4d3497af4fea9cc, entries=150, sequenceid=12, filesize=11.6 K 2023-03-26 13:54:38,450 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for a59be4541fa5742a2bd937206749175d in 103ms, sequenceid=12, compaction requested=false 2023-03-26 13:54:38,450 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-26 13:54:38,451 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:38,452 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:38,460 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:38,460 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:38,460 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,460 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:38,460 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:38,461 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,461 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:38,462 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,462 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:38,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:38,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d966814[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:38,463 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:38,464 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,472 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,472 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:38,472 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:38,472 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:38,472 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,472 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:38,474 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,474 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:38,474 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:38,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,474 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:38,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:38,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:38,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:38,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:38,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:38,476 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-03-26 13:54:38,476 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:38,476 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-26 13:54:38,476 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:38,476 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-26 13:54:38,476 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:38,476 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-26 13:54:38,484 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/499ba14fe3de436584235367cdf437eb 2023-03-26 13:54:38,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:38,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:38,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:38,490 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:38,490 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:38,490 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:38,490 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:38,490 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:38,492 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,492 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,492 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:38,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:38,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:38,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:38,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:38,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:38,495 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:38,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,497 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:38,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,499 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:38,499 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:38,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:38,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:38,499 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:38,499 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:38,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:38,499 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,500 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:38,500 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:38,500 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,500 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,500 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,500 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,500 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:38,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:38,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:38,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838938494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:38,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838938495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,501 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:38,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838938496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:38,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838938501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,505 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:38,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838938501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,600 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:38,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:38,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:38,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:38,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:38,604 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:38,604 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:38,604 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:38,604 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:38,604 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,604 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:38,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838938603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,607 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,607 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:38,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:38,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:38,607 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,607 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:38,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:38,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838938606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,608 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:38,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838938606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:38,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:38,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:38,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:38,608 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:38,609 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:38,609 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:38,609 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:38,609 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:38,609 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:38,609 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:38,609 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:38,609 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:38,609 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:38,609 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:38,609 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,609 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:38,609 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:38,610 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,610 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:38,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:38,611 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,611 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:38,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:38,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:38,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:38,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:38,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a43aac3[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:38,612 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:38,612 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,613 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,613 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:38,613 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:38,613 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:38,614 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,614 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:38,615 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,615 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:38,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:38,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:38,615 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:38,615 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:38,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:38,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:38,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:38,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:38,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:38,617 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:38,617 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:38,617 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:38,617 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:38,617 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:38,627 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,627 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:38,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:38,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,627 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,627 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:38,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:38,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:38,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:38,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:38,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,629 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:38,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,632 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:38,632 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:38,632 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,632 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,632 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:38,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:38,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:38,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:38,632 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,633 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,633 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,633 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,633 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:38,633 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:38,633 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:38,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:38,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:38,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:38,703 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:38,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838938702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:38,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838938703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,733 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:38,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:38,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:38,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:38,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:38,736 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:38,736 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:38,736 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:38,736 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:38,737 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,737 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,738 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,738 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:38,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:38,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:38,738 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,738 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:38,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:38,739 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:38,739 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:38,739 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:38,739 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:38,739 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:38,739 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:38,739 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:38,740 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:38,740 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:38,740 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:38,740 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:38,740 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:38,740 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:38,740 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,740 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:38,740 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:38,741 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,741 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:38,742 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,742 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:38,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:38,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:38,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57968af0[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:38,743 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:38,743 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,745 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,745 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:38,745 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:38,745 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:38,745 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,746 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:38,747 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,747 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:38,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:38,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:38,747 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:38,747 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:38,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:38,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:38,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:38,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:38,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:38,749 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:38,750 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:38,750 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:38,750 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:38,750 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:38,751 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,751 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,751 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:38,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:38,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:38,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:38,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:38,751 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:38,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:38,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,753 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:38,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,755 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:38,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:38,755 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:38,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:38,755 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:38,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:38,755 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:38,755 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:38,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:38,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:38,755 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,756 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,756 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,755 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:38,756 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:38,756 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,756 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,756 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,808 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:38,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838938807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:38,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838938809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:38,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838938809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,856 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:38,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:38,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:38,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:38,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:38,859 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:38,859 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:38,859 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:38,859 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:38,860 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,860 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,862 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,862 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:38,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:38,863 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,863 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:38,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:38,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:38,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:38,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:38,864 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:38,864 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:38,864 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:38,864 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:38,864 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:38,864 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:38,864 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:38,864 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:38,864 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:38,864 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:38,864 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:38,864 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,864 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:38,864 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:38,866 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,866 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:38,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:38,866 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,866 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:38,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:38,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:38,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:38,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:38,868 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:38,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@496d759e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:38,868 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,869 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,869 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:38,869 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,869 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:38,869 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:38,869 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:38,870 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:38,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:38,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:38,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:38,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:38,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:38,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:38,873 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:38,873 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:38,873 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:38,873 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:38,873 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:38,875 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,875 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:38,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:38,876 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:38,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:38,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:38,876 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:38,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:38,877 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:38,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,879 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:38,879 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:38,879 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:38,879 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:38,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:38,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:38,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:38,879 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,880 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,880 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,880 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,880 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,880 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,880 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:38,880 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:38,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:38,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:38,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:38,926 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/e480469324b940c8904a17c4b3fcf834 2023-03-26 13:54:38,980 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:38,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:38,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:38,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:38,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:38,983 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:38,983 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:38,983 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:38,983 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:38,984 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,984 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,985 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:38,985 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:38,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:38,985 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,985 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:38,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,986 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:38,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:38,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:38,986 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:38,986 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:38,986 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:38,986 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:38,987 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:38,987 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:38,987 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:38,987 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:38,987 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:38,987 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:38,987 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:38,987 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:38,987 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:38,987 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:38,987 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,987 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:38,987 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:38,988 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,988 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:38,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:38,989 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:38,989 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:38,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:38,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:38,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:38,990 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:38,990 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3745726f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:38,992 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:38,993 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:38,993 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:38,993 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:38,993 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,993 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:38,994 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,994 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:38,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:38,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:38,994 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:38,994 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:38,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:38,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:38,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:38,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:38,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:38,998 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:38,998 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:38,998 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:38,998 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:38,998 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:39,000 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,000 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:39,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,000 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:39,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:39,001 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,002 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:39,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838939004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:39,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838939005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,007 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,008 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:39,008 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:39,008 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:39,008 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:39,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:39,008 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:39,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:39,008 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:39,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:39,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:39,008 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:39,008 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:39,008 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,008 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:39,008 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:39,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:39,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:39,108 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:39,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:39,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:39,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:39,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:39,112 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:39,112 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:39,112 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:39,112 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:39,112 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:39,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838939111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,112 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,112 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:39,113 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:39,113 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:39,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:39,114 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,114 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:39,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:39,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:39,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:39,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:39,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:39,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:39,115 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:39,115 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:39,115 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:39,115 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:39,115 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:39,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:39,115 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:39,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838939114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,115 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:39,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:39,116 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:39,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838939114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,116 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:39,116 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:39,117 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,117 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:39,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:39,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:39,118 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:39,118 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:39,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:39,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6007d3f6[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:39,119 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:39,119 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:39,121 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:39,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:39,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:39,121 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,121 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:39,121 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:39,121 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:39,121 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:39,123 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,123 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:39,123 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:39,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:39,123 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:39,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:39,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:39,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:39,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:39,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:39,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:39,127 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:39,127 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:39,127 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:39,127 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:39,127 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:39,128 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,128 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:39,128 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:39,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:39,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:39,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:39,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:39,131 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:39,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:39,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:39,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:39,132 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:39,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:39,135 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:39,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:39,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:39,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:39,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:39,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:39,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:39,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:39,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:39,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:39,135 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:39,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:39,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:39,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,135 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:39,236 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:39,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:39,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:39,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:39,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:39,239 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:39,239 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:39,239 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:39,239 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:39,239 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,239 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:39,242 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:39,242 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:39,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:39,242 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,242 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:39,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:39,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:39,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:39,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:39,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:39,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:39,243 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:39,243 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:39,243 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:39,243 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:39,243 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:39,244 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:39,244 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:39,244 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:39,244 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:39,244 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:39,244 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:39,244 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,244 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:39,244 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:39,245 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:39,245 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:39,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:39,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:39,246 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:39,246 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:39,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:39,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:39,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:39,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:39,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a5140c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:39,247 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:39,247 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:39,248 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:39,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:39,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:39,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:39,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:39,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:39,249 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,249 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:39,252 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:39,252 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:39,252 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:39,258 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:39,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:39,261 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:39,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:39,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:39,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:39,261 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:39,264 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,264 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:39,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,264 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:39,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,266 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:39,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:39,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,268 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,268 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:39,268 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:39,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:39,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:39,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:39,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:39,268 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:39,268 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:39,268 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,268 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:39,268 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:39,268 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,268 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:39,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:39,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:39,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:39,269 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:39,269 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:39,336 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/96e17c02271f496d8758719958eae2af 2023-03-26 13:54:39,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/499ba14fe3de436584235367cdf437eb as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/499ba14fe3de436584235367cdf437eb 2023-03-26 13:54:39,344 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/499ba14fe3de436584235367cdf437eb, entries=200, sequenceid=50, filesize=13.9 K 2023-03-26 13:54:39,345 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/e480469324b940c8904a17c4b3fcf834 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/e480469324b940c8904a17c4b3fcf834 2023-03-26 13:54:39,349 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/e480469324b940c8904a17c4b3fcf834, entries=150, sequenceid=50, filesize=11.6 K 2023-03-26 13:54:39,349 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/96e17c02271f496d8758719958eae2af as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/96e17c02271f496d8758719958eae2af 2023-03-26 13:54:39,354 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/96e17c02271f496d8758719958eae2af, entries=150, sequenceid=50, filesize=11.6 K 2023-03-26 13:54:39,354 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for a59be4541fa5742a2bd937206749175d in 878ms, sequenceid=50, compaction requested=false 2023-03-26 13:54:39,355 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:39,369 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:39,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:39,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:39,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:39,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:39,372 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:39,372 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:39,372 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:39,372 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:39,372 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,372 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:39,373 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:39,373 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:39,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:39,374 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,374 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:39,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:39,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:39,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:39,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:39,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:39,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:39,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:39,375 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:39,375 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:39,375 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:39,375 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:39,375 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:39,375 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:39,375 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:39,375 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-26 13:54:39,375 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:39,376 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:39,376 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:39,376 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:39,376 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:39,376 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:39,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:39,511 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:39,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838879525, max timestamp=1679838879534Num uniques 859; 2023-03-26 13:54:39,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:54:39,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838879525, max timestamp=1679838879534Num uniques 859; 2023-03-26 13:54:39,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838879525, max timestamp=1679838879534Num uniques 859; 2023-03-26 13:54:39,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:54:39,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:54:39,567 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679838879546, max timestamp=1679838879566Num uniques 1018; 2023-03-26 13:54:39,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679838879546, max timestamp=1679838879566Num uniques 1018; 2023-03-26 13:54:39,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-26 13:54:39,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-26 13:54:39,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679838879546, max timestamp=1679838879566Num uniques 1018; 2023-03-26 13:54:39,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-26 13:54:39,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:39,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838939589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:39,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838939591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:39,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838939616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:39,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838939617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:39,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838939620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:39,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838939692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:39,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838939694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,783 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/f9871748f3fa4aad927edf382a2f9a63 2023-03-26 13:54:39,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:39,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838939896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:39,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:39,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838939896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,193 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/4dd812c2ed71493b94b882ba86c7e80e 2023-03-26 13:54:40,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:40,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838940201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,203 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:40,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838940201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,203 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=59 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/3ed2e056725540f68c5dea343250b886 2023-03-26 13:54:40,207 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/f9871748f3fa4aad927edf382a2f9a63 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/f9871748f3fa4aad927edf382a2f9a63 2023-03-26 13:54:40,211 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/f9871748f3fa4aad927edf382a2f9a63, entries=100, sequenceid=59, filesize=9.3 K 2023-03-26 13:54:40,212 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/4dd812c2ed71493b94b882ba86c7e80e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/4dd812c2ed71493b94b882ba86c7e80e 2023-03-26 13:54:40,216 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/4dd812c2ed71493b94b882ba86c7e80e, entries=100, sequenceid=59, filesize=9.3 K 2023-03-26 13:54:40,216 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/3ed2e056725540f68c5dea343250b886 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/3ed2e056725540f68c5dea343250b886 2023-03-26 13:54:40,220 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/3ed2e056725540f68c5dea343250b886, entries=100, sequenceid=59, filesize=9.3 K 2023-03-26 13:54:40,221 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for a59be4541fa5742a2bd937206749175d in 846ms, sequenceid=59, compaction requested=true 2023-03-26 13:54:40,221 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:40,221 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:40,221 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:40,221 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:40,221 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,221 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:40,221 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:40,223 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,223 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:40,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:40,223 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,223 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:40,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:40,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:40,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:40,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:40,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21c65063[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:40,225 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:40,225 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,228 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,228 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:40,228 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:40,228 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:40,228 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,228 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:40,229 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,229 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:40,229 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:40,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:40,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:40,229 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:40,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:40,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:40,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:40,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:40,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:40,232 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:40,232 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:40,232 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:40,232 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:40,232 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:40,233 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,233 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:40,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:40,233 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:40,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:40,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:40,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:40,234 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:40,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,235 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:40,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,237 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:40,237 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:40,237 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,237 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,238 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,238 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:40,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:40,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:40,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:40,238 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:40,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:40,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:40,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:40,238 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,238 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:40,238 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:40,238 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,238 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,338 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:40,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:40,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:40,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:40,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:40,341 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:40,342 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:40,342 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:40,342 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:40,342 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,342 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,346 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:40,346 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:40,347 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,347 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:40,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:40,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:40,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:40,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:40,348 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:40,348 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:40,348 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:40,349 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:40,349 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:40,349 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:40,349 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:40,349 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=140.89 KB heapSize=399.38 KB 2023-03-26 13:54:40,349 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:40,349 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:40,349 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:40,349 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:40,349 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:40,349 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:40,357 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/477d8465992a4b1a8afda585987b4026 2023-03-26 13:54:40,368 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/581635bc5f984498b4206e10b075c888 2023-03-26 13:54:40,377 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/ab43724a9d4c4a6bb3a8174c81e1b760 2023-03-26 13:54:40,381 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/477d8465992a4b1a8afda585987b4026 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/477d8465992a4b1a8afda585987b4026 2023-03-26 13:54:40,385 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/477d8465992a4b1a8afda585987b4026, entries=150, sequenceid=113, filesize=11.6 K 2023-03-26 13:54:40,385 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/581635bc5f984498b4206e10b075c888 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/581635bc5f984498b4206e10b075c888 2023-03-26 13:54:40,389 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/581635bc5f984498b4206e10b075c888, entries=150, sequenceid=113, filesize=11.6 K 2023-03-26 13:54:40,389 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/ab43724a9d4c4a6bb3a8174c81e1b760 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/ab43724a9d4c4a6bb3a8174c81e1b760 2023-03-26 13:54:40,394 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/ab43724a9d4c4a6bb3a8174c81e1b760, entries=150, sequenceid=113, filesize=11.6 K 2023-03-26 13:54:40,394 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~399.33 KB/408912, currentSize=0 B/0 for a59be4541fa5742a2bd937206749175d in 45ms, sequenceid=113, compaction requested=true 2023-03-26 13:54:40,394 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:40,395 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:40,395 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:40,395 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:40,395 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,395 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:40,395 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:40,397 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:40,397 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:40,397 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,397 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:40,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:40,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d0c983e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:40,399 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:40,399 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,400 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,400 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:40,400 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:40,400 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:40,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:40,401 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:40,402 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:40,402 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,402 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:40,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:40,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:40,404 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:40,405 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:40,405 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:40,405 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:40,405 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:40,408 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,408 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:40,408 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:40,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:40,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:40,409 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,411 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:40,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,413 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,413 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,413 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,413 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,413 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,413 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:40,413 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:40,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:40,413 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:40,413 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,413 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:40,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:40,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:40,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:40,413 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:40,413 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:40,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:40,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:40,514 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:40,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:40,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:40,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:40,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:40,517 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:40,517 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:40,517 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:40,517 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:40,517 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,517 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,518 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:40,518 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:40,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:40,519 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,519 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:40,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,519 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:40,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:40,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:40,520 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:40,520 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:40,520 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:40,520 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:40,520 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:40,520 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:40,520 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:40,520 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:40,520 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:40,520 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:40,520 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:40,520 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,520 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:40,520 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:40,522 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,522 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:40,522 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,522 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:40,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:40,523 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:40,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63ca740a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:40,523 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,525 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,525 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,525 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:40,525 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:40,525 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:40,526 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:40,527 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:40,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:40,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:40,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:40,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:40,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:40,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:40,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:40,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:40,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:40,529 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:40,529 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:40,529 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:40,529 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:40,529 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:40,531 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,531 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,531 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:40,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:40,532 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:40,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:40,533 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:40,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:40,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:40,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:40,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:40,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:40,535 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:40,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,536 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,536 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,536 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:40,536 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:40,536 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:40,536 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,536 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,536 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:40,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:40,632 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:54:40,633 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:40,633 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:40,633 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:40,633 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:40,633 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:40,633 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:40,636 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:40,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:40,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:40,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:40,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:40,641 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:40,641 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:40,641 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:40,641 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:40,642 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,642 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,642 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/062ad768f1474ad8a12bccfede06f58d 2023-03-26 13:54:40,644 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,644 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:40,644 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,645 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:40,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:40,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:40,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:40,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,646 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:40,646 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:40,646 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:40,646 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:40,646 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:40,646 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:40,646 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:40,647 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:40,647 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:40,647 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:40,647 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:40,647 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:40,647 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:40,647 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:40,647 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,647 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:40,647 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:40,648 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,648 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:40,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:40,648 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,649 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:40,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:40,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:40,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:40,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:40,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48e4190b[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:40,650 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:40,650 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,651 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,652 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,652 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:40,652 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:40,652 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:40,652 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:40,653 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:40,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:40,653 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:40,653 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:40,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:40,654 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:40,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:40,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:40,656 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:40,656 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:40,656 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:40,656 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:40,656 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:40,658 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,658 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:40,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,659 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:40,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:40,659 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:40,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,660 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,662 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,662 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,662 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:40,662 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:40,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:40,663 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:40,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:40,662 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,662 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:40,663 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:40,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:40,663 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,663 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:40,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:40,663 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:40,663 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:40,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:40,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838940661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838880659, max timestamp=1679838880664Num uniques 859; 2023-03-26 13:54:40,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838880659, max timestamp=1679838880664Num uniques 859; 2023-03-26 13:54:40,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:54:40,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:54:40,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838880659, max timestamp=1679838880664Num uniques 859; 2023-03-26 13:54:40,666 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:54:40,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:40,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838940693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:40,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838940694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:40,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838940708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,709 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:40,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838940708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,763 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:40,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:40,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:40,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:40,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:40,768 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:40,768 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:40,768 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:40,768 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:40,768 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,768 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:40,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838940767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,769 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,769 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:40,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:40,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:40,770 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,770 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:40,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:40,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:40,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:40,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:40,772 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:40,772 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:40,772 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:40,780 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:40,780 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:40,780 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:40,780 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:40,781 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:40,781 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:40,781 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:40,781 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:40,781 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,781 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:40,781 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:40,786 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,786 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:40,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:40,786 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,787 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:40,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:40,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:40,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:40,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:40,788 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:40,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b320d3c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:40,788 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,789 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,789 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:40,789 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:40,790 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:40,790 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,790 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:40,791 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:40,791 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,791 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:40,791 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:40,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:40,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:40,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:40,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:40,793 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:40,794 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:40,794 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:40,794 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:40,794 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:40,795 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:40,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:40,795 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,795 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:40,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:40,796 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:40,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:40,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,797 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:40,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:40,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838940796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:40,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838940798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,805 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,805 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,805 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,805 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,805 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,805 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,805 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:40,805 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:40,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:40,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:40,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:40,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:40,805 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:40,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:40,805 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,806 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:40,806 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:40,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:40,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:40,906 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:40,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:40,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:40,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:40,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:40,918 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:40,918 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:40,918 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:40,918 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:40,918 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,918 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,920 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:40,920 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:40,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:40,920 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,920 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:40,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:40,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:40,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:40,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:40,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:40,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:40,940 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:40,940 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:40,940 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:40,940 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:40,940 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:40,940 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:40,941 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:40,941 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:40,941 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,941 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:40,941 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:40,943 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:40,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:40,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:40,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@226a30c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:40,945 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:40,946 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,947 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,947 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:40,947 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:40,947 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:40,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:40,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:40,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:40,949 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:40,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:40,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:40,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:40,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:40,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:40,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:40,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:40,952 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:40,952 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:40,952 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:40,952 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:40,953 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:40,954 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,954 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:40,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:40,954 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:40,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:40,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:40,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:40,955 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:40,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:40,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,956 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:40,958 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,959 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,959 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:40,959 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:40,959 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:40,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:40,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:40,959 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:40,959 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,959 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:40,959 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:40,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:40,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:40,959 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:40,959 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:40,959 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:40,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:40,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:40,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:40,973 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:40,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838940970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:41,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838941000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:41,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838941001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,054 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/1acd1ef6fe024b8eb5959e32ff0dcbab 2023-03-26 13:54:41,059 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:41,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:41,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:41,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:41,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:41,063 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:41,063 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:41,063 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:41,063 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:41,064 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,064 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,065 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:41,065 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:41,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:41,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:41,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:41,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:41,066 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:41,066 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:41,067 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:41,067 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:41,067 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:41,067 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:41,067 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:41,067 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:41,067 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:41,067 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:41,067 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:41,067 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:41,067 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:41,067 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,067 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:41,067 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:41,070 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,070 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:41,070 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,070 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:41,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:41,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:41,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:41,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32fcfd66[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:41,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:41,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,074 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,074 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:41,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:41,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:41,075 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:41,076 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:41,076 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:41,076 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:41,076 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:41,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:41,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:41,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:41,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:41,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:41,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:41,079 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:41,079 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:41,079 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:41,079 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:41,079 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:41,081 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,081 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,081 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:41,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:41,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:41,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:41,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:41,081 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:41,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:41,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,083 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:41,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:41,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,086 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:41,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:41,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:41,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:41,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:41,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:41,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:41,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:41,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:41,087 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:41,087 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:41,187 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:41,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:41,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:41,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:41,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:41,191 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:41,191 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:41,191 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:41,191 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:41,191 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,191 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,193 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,193 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:41,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:41,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:41,193 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,193 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:41,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:41,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,194 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:41,194 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:41,194 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:41,194 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:41,194 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:41,194 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:41,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:41,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:41,195 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:41,195 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:41,195 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:41,195 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:41,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:41,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:41,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:41,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:41,197 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,197 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:41,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:41,197 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,197 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:41,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:41,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:41,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:41,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:41,199 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:41,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a389673[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:41,199 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,201 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,201 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:41,201 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:41,201 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:41,201 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,201 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:41,206 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:41,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:41,206 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:41,206 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:41,206 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:41,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:41,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:41,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:41,210 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:41,210 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:41,210 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:41,210 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:41,210 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:41,210 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:41,212 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,212 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:41,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,212 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:41,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:41,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,213 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:41,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,215 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:41,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,218 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:41,218 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:41,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:41,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:41,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:41,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:41,234 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,234 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,234 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:41,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:41,234 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:41,234 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:41,234 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:41,234 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,235 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,235 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,235 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:41,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838941275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:41,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838941304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,307 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:41,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838941305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,335 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:41,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:41,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:41,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:41,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:41,338 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:41,338 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:41,338 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:41,339 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:41,339 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,339 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,340 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,340 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:41,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:41,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:41,341 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,341 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:41,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:41,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:41,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:41,344 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:41,344 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:41,344 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:41,344 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:41,344 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:41,344 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:41,344 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:41,344 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:41,344 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:41,344 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:41,344 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:41,345 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,345 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:41,345 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:41,347 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,347 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:41,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:41,347 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,347 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:41,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:41,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:41,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:41,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:41,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11a83be5[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:41,349 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:41,349 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,350 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,351 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:41,351 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:41,351 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,351 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:41,351 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:41,352 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:41,352 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:41,352 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:41,352 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:41,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:41,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:41,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:41,354 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:41,354 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:41,354 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:41,354 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:41,354 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:41,356 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,356 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:41,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:41,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,356 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:41,356 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:41,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:41,357 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:41,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,361 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:41,361 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:41,361 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,361 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,361 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,361 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,361 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:41,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:41,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:41,361 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:41,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:41,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:41,361 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:41,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:41,362 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,362 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:41,362 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:41,396 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-26 13:54:41,462 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:41,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:41,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:41,464 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/eccb20ea57644adc9233ef9fa16359f1 2023-03-26 13:54:41,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:41,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:41,466 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:41,466 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:41,466 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:41,466 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:41,466 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,466 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,467 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:41,467 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:41,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:41,468 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,468 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:41,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:41,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:41,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:41,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:41,469 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:41,469 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:41,469 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:41,469 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:41,469 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:41,469 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:41,469 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:41,469 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/062ad768f1474ad8a12bccfede06f58d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/062ad768f1474ad8a12bccfede06f58d 2023-03-26 13:54:41,469 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:41,469 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:41,469 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:41,470 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:41,470 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,470 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:41,470 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:41,474 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,474 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:41,474 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,474 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:41,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:41,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:41,476 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/062ad768f1474ad8a12bccfede06f58d, entries=250, sequenceid=125, filesize=16.2 K 2023-03-26 13:54:41,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:41,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2621c41f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:41,476 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:41,476 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,477 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/1acd1ef6fe024b8eb5959e32ff0dcbab as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/1acd1ef6fe024b8eb5959e32ff0dcbab 2023-03-26 13:54:41,477 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,478 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:41,478 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:41,478 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:41,478 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,478 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:41,480 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:41,480 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:41,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,480 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:41,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:41,481 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/1acd1ef6fe024b8eb5959e32ff0dcbab, entries=150, sequenceid=125, filesize=11.6 K 2023-03-26 13:54:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:41,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,482 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/eccb20ea57644adc9233ef9fa16359f1 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/eccb20ea57644adc9233ef9fa16359f1 2023-03-26 13:54:41,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:41,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:41,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:41,483 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:41,483 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:41,483 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:41,483 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:41,483 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:41,485 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,485 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,485 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:41,485 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/eccb20ea57644adc9233ef9fa16359f1, entries=150, sequenceid=125, filesize=11.6 K 2023-03-26 13:54:41,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:41,485 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:41,487 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a59be4541fa5742a2bd937206749175d in 855ms, sequenceid=125, compaction requested=true 2023-03-26 13:54:41,487 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,487 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:41,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,487 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:41,487 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 13:54:41,487 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 13:54:41,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,487 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:41,487 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:41,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:41,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:41,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,489 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64106 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 13:54:41,489 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/A is initiating minor compaction (all files) 2023-03-26 13:54:41,490 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/A in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:41,490 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 13:54:41,490 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/B is initiating minor compaction (all files) 2023-03-26 13:54:41,490 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:41,490 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/32c04a2eb6594ac29a6f943e6dcaed5b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/499ba14fe3de436584235367cdf437eb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/f9871748f3fa4aad927edf382a2f9a63, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/477d8465992a4b1a8afda585987b4026, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/062ad768f1474ad8a12bccfede06f58d] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=62.6 K 2023-03-26 13:54:41,490 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:41,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:41,490 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:41,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:41,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:41,490 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/B in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:41,490 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:41,490 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6008e0a6bea7449f8625c72625dc4c36, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/e480469324b940c8904a17c4b3fcf834, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/4dd812c2ed71493b94b882ba86c7e80e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/581635bc5f984498b4206e10b075c888, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/1acd1ef6fe024b8eb5959e32ff0dcbab] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=55.7 K 2023-03-26 13:54:41,490 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:41,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:41,491 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,491 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,490 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:41,491 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:41,491 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 6008e0a6bea7449f8625c72625dc4c36, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1679838878336 2023-03-26 13:54:41,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:41,491 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,491 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 32c04a2eb6594ac29a6f943e6dcaed5b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1679838878336 2023-03-26 13:54:41,491 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e480469324b940c8904a17c4b3fcf834, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679838878388 2023-03-26 13:54:41,491 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:41,491 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,492 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 499ba14fe3de436584235367cdf437eb, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679838878388 2023-03-26 13:54:41,492 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 4dd812c2ed71493b94b882ba86c7e80e, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1679838878481 2023-03-26 13:54:41,492 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting f9871748f3fa4aad927edf382a2f9a63, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1679838878481 2023-03-26 13:54:41,492 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 581635bc5f984498b4206e10b075c888, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1679838879587 2023-03-26 13:54:41,492 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 477d8465992a4b1a8afda585987b4026, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1679838879587 2023-03-26 13:54:41,493 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 1acd1ef6fe024b8eb5959e32ff0dcbab, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1679838880627 2023-03-26 13:54:41,493 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 062ad768f1474ad8a12bccfede06f58d, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1679838880623 2023-03-26 13:54:41,501 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#B#compaction#255 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:41,504 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#A#compaction#256 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:41,509 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/6a81af28407048e0a0c95610c62700ac as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6a81af28407048e0a0c95610c62700ac 2023-03-26 13:54:41,520 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a59be4541fa5742a2bd937206749175d/B of a59be4541fa5742a2bd937206749175d into 6a81af28407048e0a0c95610c62700ac(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:41,520 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:41,520 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/B, priority=11, startTime=1679838881487; duration=0sec 2023-03-26 13:54:41,520 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:41,520 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 13:54:41,521 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 13:54:41,521 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/C is initiating minor compaction (all files) 2023-03-26 13:54:41,521 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/C in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:41,522 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/0921b04106fd4d21a4d3497af4fea9cc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/96e17c02271f496d8758719958eae2af, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/3ed2e056725540f68c5dea343250b886, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/ab43724a9d4c4a6bb3a8174c81e1b760, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/eccb20ea57644adc9233ef9fa16359f1] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=55.7 K 2023-03-26 13:54:41,522 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 0921b04106fd4d21a4d3497af4fea9cc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1679838878336 2023-03-26 13:54:41,522 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 96e17c02271f496d8758719958eae2af, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679838878388 2023-03-26 13:54:41,522 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 3ed2e056725540f68c5dea343250b886, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=59, earliestPutTs=1679838878481 2023-03-26 13:54:41,523 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting ab43724a9d4c4a6bb3a8174c81e1b760, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1679838879587 2023-03-26 13:54:41,523 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting eccb20ea57644adc9233ef9fa16359f1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1679838880627 2023-03-26 13:54:41,531 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#C#compaction#257 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:41,591 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:41,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:41,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:41,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:41,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:41,596 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:41,596 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:41,596 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:41,596 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:41,597 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,597 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,598 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:41,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:41,598 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,598 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,599 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:41,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:41,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:41,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:41,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:41,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:41,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:41,600 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:41,600 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:41,600 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:41,608 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:41,608 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:41,608 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:41,608 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:41,608 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-26 13:54:41,608 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:41,608 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:41,608 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:41,608 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:41,609 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:41,609 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:41,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:41,716 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:41,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:41,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838941730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:41,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838941731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:41,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838941782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:41,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838941810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:41,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838941810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:41,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838941833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:41,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838941834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:41,919 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/6418062110c74fbebba1da143fc12c77 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/6418062110c74fbebba1da143fc12c77 2023-03-26 13:54:41,924 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a59be4541fa5742a2bd937206749175d/A of a59be4541fa5742a2bd937206749175d into 6418062110c74fbebba1da143fc12c77(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:41,924 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:41,924 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/A, priority=11, startTime=1679838881487; duration=0sec 2023-03-26 13:54:41,924 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:41,939 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/eac81f885b594fe6a07d1be538bc716a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/eac81f885b594fe6a07d1be538bc716a 2023-03-26 13:54:41,944 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a59be4541fa5742a2bd937206749175d/C of a59be4541fa5742a2bd937206749175d into eac81f885b594fe6a07d1be538bc716a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:41,944 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:41,944 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/C, priority=11, startTime=1679838881487; duration=0sec 2023-03-26 13:54:41,944 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:42,026 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/a1e8265bd1284776850e90676a7dddb8 2023-03-26 13:54:42,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:42,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838942037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,039 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:42,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838942037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:42,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838942341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:42,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838942342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,437 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/6a7d051bac2c405c864836d2c8574185 2023-03-26 13:54:42,447 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/4c42adad613746bb855c77d7f8294287 2023-03-26 13:54:42,451 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/a1e8265bd1284776850e90676a7dddb8 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/a1e8265bd1284776850e90676a7dddb8 2023-03-26 13:54:42,456 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/a1e8265bd1284776850e90676a7dddb8, entries=150, sequenceid=162, filesize=11.8 K 2023-03-26 13:54:42,460 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/6a7d051bac2c405c864836d2c8574185 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6a7d051bac2c405c864836d2c8574185 2023-03-26 13:54:42,465 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6a7d051bac2c405c864836d2c8574185, entries=150, sequenceid=162, filesize=11.8 K 2023-03-26 13:54:42,466 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/4c42adad613746bb855c77d7f8294287 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/4c42adad613746bb855c77d7f8294287 2023-03-26 13:54:42,470 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/4c42adad613746bb855c77d7f8294287, entries=150, sequenceid=162, filesize=11.8 K 2023-03-26 13:54:42,471 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for a59be4541fa5742a2bd937206749175d in 863ms, sequenceid=162, compaction requested=false 2023-03-26 13:54:42,471 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:42,471 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:42,471 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:42,471 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:42,471 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,471 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:42,471 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:42,474 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:42,474 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:42,475 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:42,475 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:42,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:42,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:42,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:42,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:42,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:42,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:42,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73e30a70[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:42,478 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:42,478 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:42,479 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:42,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:42,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:42,479 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:42,479 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:42,479 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:42,480 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,480 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:42,481 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:42,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:42,481 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:42,481 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:42,481 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:42,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:42,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:42,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:42,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:42,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:42,485 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:42,485 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:42,485 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:42,485 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:42,485 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:42,491 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:42,491 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:42,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:42,491 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:42,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:42,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:42,492 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:42,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:42,494 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:42,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,497 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,497 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:42,497 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:42,497 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,497 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:42,497 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:42,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:42,497 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:42,497 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:42,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:42,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:42,497 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:42,497 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:42,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:42,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:42,497 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:42,498 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:42,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:42,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:42,598 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:42,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:42,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:42,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:42,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:42,602 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:42,602 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:42,602 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:42,602 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:42,602 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:42,602 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:42,604 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:42,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:42,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:42,604 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:42,604 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,605 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:42,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:42,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:42,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:42,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:42,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:42,605 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:42,605 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:42,605 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:42,606 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:42,606 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:42,606 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:42,606 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:42,606 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 13:54:42,607 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:42,607 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:42,607 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:42,607 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:42,607 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:42,607 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:42,618 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/73553a6392dd408da2dea16852d5f631 2023-03-26 13:54:42,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:42,787 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:42,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:42,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838942826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,829 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:42,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838942826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838882813, max timestamp=1679838882827Num uniques 859; 2023-03-26 13:54:42,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:54:42,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838882813, max timestamp=1679838882827Num uniques 859; 2023-03-26 13:54:42,829 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838882813, max timestamp=1679838882827Num uniques 859; 2023-03-26 13:54:42,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:54:42,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:54:42,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:42,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838942873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:42,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838942873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:42,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838942877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:42,934 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:42,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838942931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838942931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:42,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838942978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:42,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838942978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:42,982 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:42,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838942980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,029 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/0f2d23cf68dd45ed8ffac49e09cea5ae 2023-03-26 13:54:43,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:43,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838943135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:43,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838943135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:43,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838943182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:43,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838943183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:43,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838943183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,440 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=175 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/15702b95af774f16ae59bb8e5fa4d72a 2023-03-26 13:54:43,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:43,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838943440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:43,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838943440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,449 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/73553a6392dd408da2dea16852d5f631 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/73553a6392dd408da2dea16852d5f631 2023-03-26 13:54:43,454 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/73553a6392dd408da2dea16852d5f631, entries=150, sequenceid=175, filesize=11.8 K 2023-03-26 13:54:43,455 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/0f2d23cf68dd45ed8ffac49e09cea5ae as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/0f2d23cf68dd45ed8ffac49e09cea5ae 2023-03-26 13:54:43,459 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/0f2d23cf68dd45ed8ffac49e09cea5ae, entries=150, sequenceid=175, filesize=11.8 K 2023-03-26 13:54:43,460 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/15702b95af774f16ae59bb8e5fa4d72a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/15702b95af774f16ae59bb8e5fa4d72a 2023-03-26 13:54:43,463 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/15702b95af774f16ae59bb8e5fa4d72a, entries=150, sequenceid=175, filesize=11.8 K 2023-03-26 13:54:43,464 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a59be4541fa5742a2bd937206749175d in 858ms, sequenceid=175, compaction requested=true 2023-03-26 13:54:43,464 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:43,464 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:43,464 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:43,464 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:43,464 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,464 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:43,464 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:43,466 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,466 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:43,467 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,467 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:43,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:43,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:43,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d768171[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:43,469 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:43,469 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,470 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,470 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:43,470 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:43,470 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:43,472 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:43,472 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:43,472 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:43,472 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,472 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:43,473 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:43,473 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:43,473 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:43,473 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:43,473 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:43,473 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:43,475 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,475 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:43,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:43,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,476 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:43,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:43,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:43,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:43,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,476 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:43,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,478 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:43,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:43,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:43,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:43,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:43,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,480 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:43,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:43,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:43,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,481 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:43,481 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:43,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:43,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:43,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:43,487 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-26 13:54:43,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:43,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:43,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:43,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:43,488 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:43,488 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:43,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:43,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838943497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,502 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:43,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838943498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:43,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838943498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,506 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/8d7063a83c104366aa6b18c2adeffaac 2023-03-26 13:54:43,581 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:43,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:43,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:43,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:43,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:43,585 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:43,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:43,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:43,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:43,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,587 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:43,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:43,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:43,587 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,587 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:43,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:43,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:43,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:43,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:43,589 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:43,589 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:43,589 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:43,589 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:43,591 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,591 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:43,591 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,591 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:43,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:43,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:43,593 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:43,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d51e829[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:43,593 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,595 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,595 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:43,595 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:43,595 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:43,595 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,595 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:43,597 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:43,597 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:43,597 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,597 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:43,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:43,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:43,599 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:43,599 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:43,599 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:43,599 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:43,599 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:43,600 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,600 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:43,600 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:43,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:43,601 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,602 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:43,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838943603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:43,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838943604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,605 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,605 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,605 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,605 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,605 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:43,605 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:43,605 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:43,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:43,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:43,605 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,606 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,605 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:43,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:43,606 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:43,606 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:43,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:43,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:43,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:43,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838943604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,706 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:43,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:43,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:43,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:43,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:43,712 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:43,712 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:43,712 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:43,712 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:43,712 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,712 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,715 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:43,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:43,715 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:43,715 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,715 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:43,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:43,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,717 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:43,717 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:43,717 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:43,717 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:43,717 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:43,717 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:43,717 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:43,717 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:43,717 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:43,717 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:43,718 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:43,718 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:43,718 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:43,718 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:43,718 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,718 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:43,718 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:43,719 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,719 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:43,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:43,719 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,719 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:43,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:43,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:43,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:43,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:43,721 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:43,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15f8d2a0[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:43,721 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,722 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,722 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:43,722 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:43,722 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:43,723 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,723 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:43,724 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:43,724 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:43,724 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:43,724 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:43,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:43,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:43,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:43,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:43,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:43,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:43,727 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:43,727 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:43,727 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:43,727 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:43,727 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:43,729 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,729 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:43,729 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:43,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:43,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:43,730 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,731 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:43,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,737 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,737 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,737 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:43,738 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:43,737 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:43,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:43,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:43,737 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:43,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:43,738 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:43,738 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:43,738 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,738 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:43,738 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,738 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:43,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:43,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:43,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838943805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:43,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838943806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,809 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:43,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838943808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,838 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:43,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:43,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:43,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:43,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:43,842 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:43,842 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:43,842 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:43,842 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:43,842 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,842 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,843 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:43,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:43,843 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,844 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,844 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:43,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:43,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:43,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:43,845 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:43,845 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:43,845 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:43,845 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:43,845 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:43,845 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:43,845 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:43,845 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:43,845 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:43,845 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:43,845 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:43,845 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,845 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:43,845 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:43,852 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,852 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:43,853 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,853 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:43,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:43,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:43,854 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:43,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e23fae5[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:43,854 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,855 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:43,855 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:43,855 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:43,855 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,855 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:43,856 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,857 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:43,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:43,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:43,857 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:43,857 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:43,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:43,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:43,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:43,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:43,859 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:43,859 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:43,859 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:43,859 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:43,859 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:43,860 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,860 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,860 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:43,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:43,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:43,861 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:43,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:43,862 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:43,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:43,865 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,865 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,865 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,865 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,865 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:43,865 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:43,865 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:43,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:43,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:43,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:43,865 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:43,866 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,865 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:43,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:43,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:43,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:43,866 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:43,866 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:43,938 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/9e7979728998427f8ce1c1f0d20a46a1 2023-03-26 13:54:43,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:43,947 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:43,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838943944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838943944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,948 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/18a787d626d54e2fac78910fa620d2dc 2023-03-26 13:54:43,951 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/8d7063a83c104366aa6b18c2adeffaac as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/8d7063a83c104366aa6b18c2adeffaac 2023-03-26 13:54:43,955 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/8d7063a83c104366aa6b18c2adeffaac, entries=200, sequenceid=212, filesize=14.1 K 2023-03-26 13:54:43,956 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/9e7979728998427f8ce1c1f0d20a46a1 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/9e7979728998427f8ce1c1f0d20a46a1 2023-03-26 13:54:43,959 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/9e7979728998427f8ce1c1f0d20a46a1, entries=150, sequenceid=212, filesize=11.8 K 2023-03-26 13:54:43,960 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/18a787d626d54e2fac78910fa620d2dc as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/18a787d626d54e2fac78910fa620d2dc 2023-03-26 13:54:43,963 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/18a787d626d54e2fac78910fa620d2dc, entries=150, sequenceid=212, filesize=11.8 K 2023-03-26 13:54:43,964 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for a59be4541fa5742a2bd937206749175d in 477ms, sequenceid=212, compaction requested=true 2023-03-26 13:54:43,964 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:43,964 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:43,964 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:43,964 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:54:43,964 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:54:43,964 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:43,965 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48166 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:54:43,965 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/B is initiating minor compaction (all files) 2023-03-26 13:54:43,965 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/B in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:43,965 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6a81af28407048e0a0c95610c62700ac, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6a7d051bac2c405c864836d2c8574185, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/0f2d23cf68dd45ed8ffac49e09cea5ae, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/9e7979728998427f8ce1c1f0d20a46a1] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=47.0 K 2023-03-26 13:54:43,965 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50556 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:54:43,966 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/A is initiating minor compaction (all files) 2023-03-26 13:54:43,966 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/A in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:43,966 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/6418062110c74fbebba1da143fc12c77, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/a1e8265bd1284776850e90676a7dddb8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/73553a6392dd408da2dea16852d5f631, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/8d7063a83c104366aa6b18c2adeffaac] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=49.4 K 2023-03-26 13:54:43,966 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 6a81af28407048e0a0c95610c62700ac, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1679838880627 2023-03-26 13:54:43,966 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 6418062110c74fbebba1da143fc12c77, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1679838880627 2023-03-26 13:54:43,966 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:43,967 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 6a7d051bac2c405c864836d2c8574185, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1679838880687 2023-03-26 13:54:43,967 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting a1e8265bd1284776850e90676a7dddb8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1679838880687 2023-03-26 13:54:43,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:43,967 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 0f2d23cf68dd45ed8ffac49e09cea5ae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1679838881724 2023-03-26 13:54:43,967 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 73553a6392dd408da2dea16852d5f631, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1679838881724 2023-03-26 13:54:43,967 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 9e7979728998427f8ce1c1f0d20a46a1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1679838882871 2023-03-26 13:54:43,968 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 8d7063a83c104366aa6b18c2adeffaac, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1679838882864 2023-03-26 13:54:43,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:43,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:43,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:43,971 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:43,971 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:43,971 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:43,971 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:43,971 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,972 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,973 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:43,973 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:43,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:43,973 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:43,973 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:43,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:43,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:43,975 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:43,975 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:43,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:43,975 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:43,975 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:43,975 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:43,976 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:43,976 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:43,976 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:43,976 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:43,976 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:54:43,976 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:43,977 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:43,977 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:43,977 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:43,977 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:43,977 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:43,979 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#B#compaction#267 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:43,981 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#A#compaction#268 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:44,004 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/6ae2f04015ae46a29cc980c34dda9662 2023-03-26 13:54:44,006 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/b1700eb62f4a4598876c6eb2698fdc99 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/b1700eb62f4a4598876c6eb2698fdc99 2023-03-26 13:54:44,006 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/e8aeda416fc1402a9be17ce377a98e2c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/e8aeda416fc1402a9be17ce377a98e2c 2023-03-26 13:54:44,012 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a59be4541fa5742a2bd937206749175d/B of a59be4541fa5742a2bd937206749175d into b1700eb62f4a4598876c6eb2698fdc99(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:44,012 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a59be4541fa5742a2bd937206749175d/A of a59be4541fa5742a2bd937206749175d into e8aeda416fc1402a9be17ce377a98e2c(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:44,012 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:44,012 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/A, priority=12, startTime=1679838883964; duration=0sec 2023-03-26 13:54:44,012 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:44,012 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/B, priority=12, startTime=1679838883964; duration=0sec 2023-03-26 13:54:44,012 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:44,012 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:44,012 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:54:44,014 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48166 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:54:44,014 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/C is initiating minor compaction (all files) 2023-03-26 13:54:44,014 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/C in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:44,014 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/eac81f885b594fe6a07d1be538bc716a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/4c42adad613746bb855c77d7f8294287, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/15702b95af774f16ae59bb8e5fa4d72a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/18a787d626d54e2fac78910fa620d2dc] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=47.0 K 2023-03-26 13:54:44,015 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting eac81f885b594fe6a07d1be538bc716a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1679838880627 2023-03-26 13:54:44,015 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 4c42adad613746bb855c77d7f8294287, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1679838880687 2023-03-26 13:54:44,015 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 15702b95af774f16ae59bb8e5fa4d72a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=175, earliestPutTs=1679838881724 2023-03-26 13:54:44,015 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 18a787d626d54e2fac78910fa620d2dc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1679838882871 2023-03-26 13:54:44,028 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/88f34535896d42deb59da0ef2d537894 2023-03-26 13:54:44,030 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#C#compaction#271 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:44,057 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=222 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/7790a4c70cca45e5ac2f5e3b452dbc66 2023-03-26 13:54:44,061 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/6ae2f04015ae46a29cc980c34dda9662 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/6ae2f04015ae46a29cc980c34dda9662 2023-03-26 13:54:44,064 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/6ae2f04015ae46a29cc980c34dda9662, entries=150, sequenceid=222, filesize=11.8 K 2023-03-26 13:54:44,065 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/88f34535896d42deb59da0ef2d537894 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/88f34535896d42deb59da0ef2d537894 2023-03-26 13:54:44,068 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/88f34535896d42deb59da0ef2d537894, entries=150, sequenceid=222, filesize=11.8 K 2023-03-26 13:54:44,069 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/7790a4c70cca45e5ac2f5e3b452dbc66 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/7790a4c70cca45e5ac2f5e3b452dbc66 2023-03-26 13:54:44,072 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/7790a4c70cca45e5ac2f5e3b452dbc66, entries=150, sequenceid=222, filesize=11.8 K 2023-03-26 13:54:44,073 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for a59be4541fa5742a2bd937206749175d in 97ms, sequenceid=222, compaction requested=false 2023-03-26 13:54:44,073 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:44,073 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:44,073 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:44,073 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:44,073 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,073 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:44,073 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:44,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,075 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,075 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:44,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:44,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27879159[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:44,077 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:44,077 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:44,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:44,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:44,078 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,078 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:44,080 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:44,080 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,080 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:44,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,080 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:44,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:44,083 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:44,083 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:44,083 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:44,083 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:44,083 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:44,084 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,084 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,084 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:44,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:44,085 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:44,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,086 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:44,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:44,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:44,089 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:44,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:44,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:44,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:44,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:44,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:44,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:44,089 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:44,089 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:44,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:44,116 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:54:44,117 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:44,117 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:44,117 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:44,117 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:44,117 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:44,117 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:44,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838884129, max timestamp=1679838884140Num uniques 859; 2023-03-26 13:54:44,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:54:44,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838884129, max timestamp=1679838884140Num uniques 859; 2023-03-26 13:54:44,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838884129, max timestamp=1679838884140Num uniques 859; 2023-03-26 13:54:44,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:54:44,142 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:54:44,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:44,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838944160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:44,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838944161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:44,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838944162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,190 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:44,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:44,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:44,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:44,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:44,192 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:44,192 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:44,192 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:44,193 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:44,193 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,193 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,194 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:44,194 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:44,194 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,195 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:44,195 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:44,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:44,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:44,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:44,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:44,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:44,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:44,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:44,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:44,196 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:44,196 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:44,196 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:44,196 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:44,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:44,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:44,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:44,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:44,198 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,198 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,199 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,199 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:44,200 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:44,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15252520[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:44,200 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,201 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,202 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:44,202 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:44,202 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:44,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:44,203 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:44,203 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,203 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:44,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,203 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:44,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:44,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:44,205 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:44,205 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:44,206 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:44,206 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:44,206 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:44,207 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,207 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,207 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:44,208 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,209 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,211 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,211 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:44,211 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:44,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:44,211 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,212 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,212 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,211 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:44,211 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:44,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:44,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:44,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:44,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:44,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:44,212 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,212 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,212 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:44,212 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:44,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:44,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838944264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:44,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:44,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838944266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838944266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,312 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:44,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:44,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:44,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:44,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:44,316 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:44,316 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:44,316 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:44,316 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:44,316 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,316 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,320 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:44,320 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:44,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:44,320 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,320 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:44,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:44,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,321 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:44,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:44,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:44,321 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:44,321 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:44,321 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:44,322 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:44,322 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:44,322 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:44,322 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:44,322 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:44,322 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:44,322 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:44,322 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:44,322 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,322 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:44,322 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:44,324 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:44,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:44,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@461b88f6[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:44,329 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:44,329 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,332 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,332 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:44,332 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:44,332 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:44,332 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,332 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:44,333 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,333 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:44,334 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:44,334 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:44,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:44,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:44,336 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:44,336 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:44,336 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:44,336 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:44,336 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:44,338 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,338 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,338 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:44,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:44,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,339 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,340 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,342 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,342 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:44,342 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:44,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:44,342 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,343 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,343 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,343 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,343 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:44,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:44,343 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:44,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:44,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:44,343 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,343 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:44,343 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:44,443 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:44,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:44,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:44,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:44,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:44,448 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:44,448 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:44,448 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:44,448 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:44,449 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,449 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,451 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/4b6e1775206e41d29de269601dbb2d38 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/4b6e1775206e41d29de269601dbb2d38 2023-03-26 13:54:44,451 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:44,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:44,451 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:44,452 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,452 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:44,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:44,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:44,453 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:44,453 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:44,453 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:44,453 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:44,453 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:44,454 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:44,454 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:44,454 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:44,454 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:44,454 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:44,454 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:44,454 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:44,454 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:44,454 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,454 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:44,454 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:44,455 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a59be4541fa5742a2bd937206749175d/C of a59be4541fa5742a2bd937206749175d into 4b6e1775206e41d29de269601dbb2d38(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:44,455 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:44,456 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/C, priority=12, startTime=1679838883964; duration=0sec 2023-03-26 13:54:44,456 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:44,458 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,458 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,458 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,458 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:44,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:44,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@270eb602[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:44,460 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:44,460 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,461 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,461 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:44,461 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:44,461 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:44,461 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,461 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:44,462 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:44,463 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:44,463 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,463 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:44,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:44,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:44,466 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:44,466 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:44,466 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:44,466 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:44,466 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:44,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:44,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838944466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,469 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:44,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838944469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:44,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838944473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,474 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,474 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,474 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:44,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,475 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,477 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,482 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:44,482 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:44,482 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,483 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,483 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,483 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,483 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,483 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,483 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:44,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:44,483 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:44,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:44,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:44,483 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:44,483 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:44,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:44,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:44,524 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/0a5f7858a28c4da98845722d913309df 2023-03-26 13:54:44,583 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:44,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:44,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:44,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:44,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:44,587 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:44,587 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:44,587 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:44,587 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:44,587 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,587 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,589 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:44,589 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:44,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:44,589 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,589 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:44,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:44,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:44,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:44,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:44,590 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:44,590 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:44,590 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:44,590 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:44,590 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:44,590 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:44,590 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:44,590 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:44,590 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:44,590 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:44,590 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:44,590 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,591 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:44,591 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:44,592 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,592 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,592 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,592 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:44,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:44,594 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:44,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c18e6e3[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:44,594 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,596 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,596 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:44,596 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:44,596 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:44,596 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,597 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:44,598 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:44,598 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:44,598 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,598 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:44,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:44,600 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:44,600 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:44,600 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:44,600 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:44,600 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:44,602 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,602 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,602 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:44,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:44,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:44,603 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,604 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,606 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,606 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,606 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:44,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:44,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:44,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:44,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:44,607 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:44,606 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,607 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:44,607 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:44,606 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:44,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:44,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:44,607 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,607 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,607 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,607 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,707 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:44,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:44,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:44,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:44,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:44,710 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:44,710 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:44,710 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:44,710 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:44,711 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,711 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,713 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:44,713 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:44,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:44,714 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,714 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:44,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:44,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:44,715 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:44,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:44,715 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:44,715 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:44,715 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:44,715 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:44,716 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:44,720 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,720 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,720 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,720 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:44,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:44,722 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:44,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a8c865[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:44,722 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,723 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,723 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:44,723 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:44,723 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:44,723 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,723 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:44,724 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,724 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,724 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:44,725 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:44,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:44,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:44,731 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:44,731 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:44,731 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:44,731 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:44,731 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:44,735 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,735 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:44,735 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:44,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:44,737 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,738 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,741 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:44,741 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:44,741 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,741 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,741 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,741 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,741 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,741 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,741 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,741 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:44,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:44,741 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:44,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:44,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:44,741 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:44,741 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:44,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:44,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:44,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:44,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838944772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:44,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838944772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:44,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838944775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,841 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:44,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:44,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:44,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:44,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:44,845 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:44,845 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:44,845 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:44,845 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:44,846 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,846 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,847 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:44,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:44,847 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,847 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:44,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:44,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,848 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:44,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:44,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:44,849 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:44,849 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:44,849 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:44,853 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:44,853 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:44,853 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:44,853 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:44,853 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:44,853 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:44,853 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:44,854 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:44,854 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,854 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:44,854 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:44,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,855 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,855 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,856 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:44,857 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:44,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aeebcbe[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:44,857 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,859 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,859 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:44,859 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:44,859 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:44,859 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,859 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:44,861 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,861 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:44,861 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,861 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:44,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:44,863 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:44,863 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:44,863 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:44,863 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:44,863 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:44,864 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,864 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,864 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:44,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:44,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:44,865 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:44,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,866 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:44,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,868 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,868 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,868 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,868 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,868 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:44,868 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:44,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:44,869 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:44,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:44,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:44,868 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:44,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:44,869 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:44,869 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,869 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:44,869 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:44,935 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/72c51e94e25f440e8e8537cdced41a44 2023-03-26 13:54:44,952 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/19c9379d43ec49f589ffae189e7ed5b5 2023-03-26 13:54:44,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:44,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838944951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:44,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838944954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,962 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/0a5f7858a28c4da98845722d913309df as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/0a5f7858a28c4da98845722d913309df 2023-03-26 13:54:44,966 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/0a5f7858a28c4da98845722d913309df, entries=150, sequenceid=235, filesize=11.8 K 2023-03-26 13:54:44,967 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/72c51e94e25f440e8e8537cdced41a44 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/72c51e94e25f440e8e8537cdced41a44 2023-03-26 13:54:44,969 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:44,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:44,971 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/72c51e94e25f440e8e8537cdced41a44, entries=150, sequenceid=235, filesize=11.8 K 2023-03-26 13:54:44,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:44,972 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/19c9379d43ec49f589ffae189e7ed5b5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/19c9379d43ec49f589ffae189e7ed5b5 2023-03-26 13:54:44,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:44,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:44,973 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:44,973 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:44,973 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:44,973 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:44,974 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,974 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,977 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:44,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:44,977 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:44,977 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/19c9379d43ec49f589ffae189e7ed5b5, entries=150, sequenceid=235, filesize=11.8 K 2023-03-26 13:54:44,977 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:44,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,977 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:44,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:44,978 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a59be4541fa5742a2bd937206749175d in 862ms, sequenceid=235, compaction requested=true 2023-03-26 13:54:44,978 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:44,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:44,978 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:44,978 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:44,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:44,978 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:44,978 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:44,978 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:44,978 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:44,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:44,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:44,979 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:44,979 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:44,979 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:44,979 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:44,979 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:44,979 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/A is initiating minor compaction (all files) 2023-03-26 13:54:44,979 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/B is initiating minor compaction (all files) 2023-03-26 13:54:44,980 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/A in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:44,980 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-26 13:54:44,980 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/e8aeda416fc1402a9be17ce377a98e2c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/6ae2f04015ae46a29cc980c34dda9662, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/0a5f7858a28c4da98845722d913309df] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=35.6 K 2023-03-26 13:54:44,980 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/B in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:44,980 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/b1700eb62f4a4598876c6eb2698fdc99, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/88f34535896d42deb59da0ef2d537894, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/72c51e94e25f440e8e8537cdced41a44] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=35.6 K 2023-03-26 13:54:44,981 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting e8aeda416fc1402a9be17ce377a98e2c, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1679838882871 2023-03-26 13:54:44,981 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:44,981 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting b1700eb62f4a4598876c6eb2698fdc99, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1679838882871 2023-03-26 13:54:44,981 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:44,981 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:44,981 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:44,981 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:44,981 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:44,981 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 6ae2f04015ae46a29cc980c34dda9662, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1679838883489 2023-03-26 13:54:44,981 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 88f34535896d42deb59da0ef2d537894, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1679838883489 2023-03-26 13:54:44,981 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 0a5f7858a28c4da98845722d913309df, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1679838884113 2023-03-26 13:54:44,982 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 72c51e94e25f440e8e8537cdced41a44, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1679838884113 2023-03-26 13:54:45,005 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#A#compaction#277 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:45,007 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#B#compaction#278 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:45,016 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/e2ec04acde86469ca59d3393ee32bc13 2023-03-26 13:54:45,029 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/121e11a209894c069ffc83f2c863a1c8 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/121e11a209894c069ffc83f2c863a1c8 2023-03-26 13:54:45,034 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/abef16f9ca534e77b417bb184248502f 2023-03-26 13:54:45,038 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a59be4541fa5742a2bd937206749175d/A of a59be4541fa5742a2bd937206749175d into 121e11a209894c069ffc83f2c863a1c8(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:45,038 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:45,038 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/A, priority=13, startTime=1679838884978; duration=0sec 2023-03-26 13:54:45,038 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:45,038 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:45,039 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:45,040 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/C is initiating minor compaction (all files) 2023-03-26 13:54:45,040 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/C in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:45,040 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/4b6e1775206e41d29de269601dbb2d38, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/7790a4c70cca45e5ac2f5e3b452dbc66, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/19c9379d43ec49f589ffae189e7ed5b5] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=35.6 K 2023-03-26 13:54:45,040 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 4b6e1775206e41d29de269601dbb2d38, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1679838882871 2023-03-26 13:54:45,041 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 7790a4c70cca45e5ac2f5e3b452dbc66, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=222, earliestPutTs=1679838883489 2023-03-26 13:54:45,041 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 19c9379d43ec49f589ffae189e7ed5b5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1679838884113 2023-03-26 13:54:45,048 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/7af8ede8d1784c8a904e117af2dbcdcf 2023-03-26 13:54:45,049 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#C#compaction#281 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:45,053 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/e2ec04acde86469ca59d3393ee32bc13 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/e2ec04acde86469ca59d3393ee32bc13 2023-03-26 13:54:45,060 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/e2ec04acde86469ca59d3393ee32bc13, entries=150, sequenceid=270, filesize=11.9 K 2023-03-26 13:54:45,061 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/abef16f9ca534e77b417bb184248502f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/abef16f9ca534e77b417bb184248502f 2023-03-26 13:54:45,065 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/abef16f9ca534e77b417bb184248502f, entries=150, sequenceid=270, filesize=11.9 K 2023-03-26 13:54:45,066 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/7af8ede8d1784c8a904e117af2dbcdcf as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/7af8ede8d1784c8a904e117af2dbcdcf 2023-03-26 13:54:45,070 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/7af8ede8d1784c8a904e117af2dbcdcf, entries=150, sequenceid=270, filesize=11.9 K 2023-03-26 13:54:45,071 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=0 B/0 for a59be4541fa5742a2bd937206749175d in 91ms, sequenceid=270, compaction requested=false 2023-03-26 13:54:45,071 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:45,071 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:45,071 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:45,071 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:45,071 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,071 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:45,071 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:45,073 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,073 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:45,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:45,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:45,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:45,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:45,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:45,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:45,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4802b747[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:45,076 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:45,076 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,077 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:45,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:45,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:45,078 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,078 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:45,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:45,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:45,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:45,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:45,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:45,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:45,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:45,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:45,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:45,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:45,082 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:45,082 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:45,082 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:45,082 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:45,082 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:45,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:45,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:45,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:45,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:45,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:45,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:45,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,087 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:45,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,088 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:45,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,090 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,091 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,091 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,091 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:45,091 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:45,091 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:45,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:45,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:45,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:45,091 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:45,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:45,091 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:45,091 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:45,091 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:45,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:45,091 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,091 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,191 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:45,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:45,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:45,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:45,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:45,195 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:45,195 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:45,195 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:45,195 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:45,195 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,195 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,196 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:45,196 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:45,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:45,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,197 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,197 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:45,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:45,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:45,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:45,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:45,198 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:45,198 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:45,198 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:45,198 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:45,198 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:45,198 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:45,198 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:45,198 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:45,198 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:45,199 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:45,199 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:45,199 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,199 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:45,199 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:45,200 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,200 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:45,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:45,201 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,201 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:45,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:45,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:45,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:45,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:45,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:45,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33d67f5a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:45,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,204 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,205 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:45,205 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:45,205 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:45,205 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,205 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:45,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,208 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:45,208 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:45,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:45,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:45,208 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:45,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:45,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:45,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:45,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:45,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:45,211 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:45,211 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:45,211 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:45,211 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:45,211 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:45,212 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,212 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,212 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:45,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:45,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:45,213 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:45,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:45,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:45,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:45,214 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:45,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,216 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,216 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:45,216 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:45,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:45,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:45,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:45,216 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:45,216 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,217 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:45,216 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:45,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:45,217 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:45,217 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:45,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:45,217 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,217 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,217 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:45,283 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 13:54:45,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:45,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:45,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:45,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:45,285 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:45,285 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:45,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:45,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838945301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:45,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838945302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:45,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838945303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,317 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:45,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:45,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:45,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:45,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:45,320 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:45,320 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:45,320 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:45,320 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:45,320 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,320 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:45,323 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:45,323 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,323 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:45,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:45,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:45,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:45,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:45,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:45,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:45,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:45,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:45,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:45,324 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:45,325 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:45,325 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:45,325 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:45,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:45,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:45,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:45,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:45,326 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,326 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:45,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:45,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:45,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:45,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:45,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:45,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:45,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75fb72a5[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:45,328 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:45,328 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,329 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,330 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:45,330 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:45,330 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:45,330 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,330 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:45,331 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:45,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:45,331 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:45,331 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:45,331 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:45,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:45,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:45,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:45,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:45,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:45,333 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:45,333 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:45,333 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:45,334 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:45,334 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:45,336 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,336 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:45,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:45,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,336 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:45,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:45,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:45,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:45,344 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,345 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:45,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,348 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,348 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,348 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,348 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,348 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,348 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,349 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,348 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:45,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:45,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:45,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:45,349 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:45,349 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:45,351 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:45,351 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:45,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:45,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:45,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:45,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838945404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:45,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838945406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,408 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:45,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838945406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,435 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/b60a00807b874fb9a3dc1422af60a58e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/b60a00807b874fb9a3dc1422af60a58e 2023-03-26 13:54:45,439 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a59be4541fa5742a2bd937206749175d/B of a59be4541fa5742a2bd937206749175d into b60a00807b874fb9a3dc1422af60a58e(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:45,439 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:45,439 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/B, priority=13, startTime=1679838884978; duration=0sec 2023-03-26 13:54:45,439 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:45,449 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:45,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:45,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:45,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:45,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:45,452 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:45,452 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:45,453 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:45,453 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:45,453 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,453 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,454 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:45,454 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:45,455 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,455 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:45,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:45,460 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:45,460 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:45,460 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/42bb6fcc3f1e444883b8f85bddbead4a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/42bb6fcc3f1e444883b8f85bddbead4a 2023-03-26 13:54:45,460 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:45,460 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:45,460 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:45,464 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a59be4541fa5742a2bd937206749175d/C of a59be4541fa5742a2bd937206749175d into 42bb6fcc3f1e444883b8f85bddbead4a(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:45,464 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:45,464 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/C, priority=13, startTime=1679838884978; duration=0sec 2023-03-26 13:54:45,465 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:45,468 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:45,468 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:45,468 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:45,468 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:45,468 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:45,468 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:45,468 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:45,468 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:45,468 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,471 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:45,471 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:45,473 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,473 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:45,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:45,474 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:45,474 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:45,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:45,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:45,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:45,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a1f2be8[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:45,475 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:45,475 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,478 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,478 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:45,478 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:45,478 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:45,478 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,478 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:45,480 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:45,480 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:45,480 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:45,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:45,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:45,484 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:45,484 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:45,484 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:45,484 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:45,484 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:45,486 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,486 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:45,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,486 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:45,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:45,486 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:45,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:45,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:45,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:45,488 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:45,490 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,490 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,490 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,490 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:45,490 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:45,490 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:45,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:45,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:45,490 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,491 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,491 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,491 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,491 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:45,491 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:45,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:45,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:45,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:45,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:45,492 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:45,591 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:45,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:45,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:45,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:45,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:45,594 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:45,594 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:45,594 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:45,594 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:45,595 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,595 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,602 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,602 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:45,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:45,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:45,602 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,602 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:45,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:45,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:45,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:45,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:45,603 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:45,603 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:45,603 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:45,604 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:45,604 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:45,604 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:45,604 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:45,604 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:45,604 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:45,604 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:45,604 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:45,604 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,604 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:45,604 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:45,606 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,606 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:45,606 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,606 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:45,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:45,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:45,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@477a1f63[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:45,607 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:45,607 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,608 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,608 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:45,608 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:45,609 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:45,609 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,609 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:45,610 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:45,610 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:45,610 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:45,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:45,610 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:45,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:45,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:45,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838945608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:45,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:45,611 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:45,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838945610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:45,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838945610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:45,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:45,612 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:45,612 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:45,612 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:45,612 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:45,612 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:45,617 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,617 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:45,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,617 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:45,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:45,617 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:45,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,618 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:45,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,620 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:45,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:45,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:45,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:45,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:45,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:45,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:45,621 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:45,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,621 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:45,621 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:45,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,621 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:45,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:45,702 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/b72fa16a9aa34577b4822f0b64020f33 2023-03-26 13:54:45,721 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:45,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:45,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:45,724 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/e499f404792240a5859dd19e5976f4c8 2023-03-26 13:54:45,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:45,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:45,725 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:45,725 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:45,725 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:45,725 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:45,726 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,726 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,727 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:45,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:45,727 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,728 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,728 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:45,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:45,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:45,731 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:45,731 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:45,731 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:45,731 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:45,731 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:45,731 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:45,731 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:45,732 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:45,732 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:45,732 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:45,732 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:45,732 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,732 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:45,732 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:45,736 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,736 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:45,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:45,736 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,736 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:45,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:45,737 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/5458206b89154c32bdfdad871206592f 2023-03-26 13:54:45,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:45,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:45,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:45,739 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:45,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47d68479[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:45,739 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,740 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:45,740 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:45,740 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/b72fa16a9aa34577b4822f0b64020f33 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/b72fa16a9aa34577b4822f0b64020f33 2023-03-26 13:54:45,741 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,741 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:45,741 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,742 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:45,742 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:45,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:45,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:45,742 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:45,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:45,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:45,744 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/b72fa16a9aa34577b4822f0b64020f33, entries=150, sequenceid=284, filesize=11.9 K 2023-03-26 13:54:45,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:45,745 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/e499f404792240a5859dd19e5976f4c8 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/e499f404792240a5859dd19e5976f4c8 2023-03-26 13:54:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:45,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:45,746 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:45,746 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:45,746 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:45,746 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:45,746 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:45,747 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,747 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,747 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:45,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:45,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:45,748 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/e499f404792240a5859dd19e5976f4c8, entries=150, sequenceid=284, filesize=11.9 K 2023-03-26 13:54:45,748 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:45,749 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/5458206b89154c32bdfdad871206592f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/5458206b89154c32bdfdad871206592f 2023-03-26 13:54:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:45,751 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/5458206b89154c32bdfdad871206592f, entries=150, sequenceid=284, filesize=11.9 K 2023-03-26 13:54:45,754 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a59be4541fa5742a2bd937206749175d in 470ms, sequenceid=284, compaction requested=true 2023-03-26 13:54:45,754 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:45,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:45,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:45,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:45,754 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:45,754 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:45,754 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:45,754 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:45,754 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,754 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:45,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:45,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:45,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:45,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:45,754 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,755 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,755 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,755 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,755 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:45,755 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,755 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:45,755 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:45,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:45,755 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:45,755 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:45,755 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/A is initiating minor compaction (all files) 2023-03-26 13:54:45,755 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/B is initiating minor compaction (all files) 2023-03-26 13:54:45,755 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/A in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:45,755 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/B in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:45,755 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/121e11a209894c069ffc83f2c863a1c8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/e2ec04acde86469ca59d3393ee32bc13, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/b72fa16a9aa34577b4822f0b64020f33] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=36.0 K 2023-03-26 13:54:45,755 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/b60a00807b874fb9a3dc1422af60a58e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/abef16f9ca534e77b417bb184248502f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/e499f404792240a5859dd19e5976f4c8] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=36.0 K 2023-03-26 13:54:45,756 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 121e11a209894c069ffc83f2c863a1c8, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1679838884113 2023-03-26 13:54:45,756 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting b60a00807b874fb9a3dc1422af60a58e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1679838884113 2023-03-26 13:54:45,756 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting e2ec04acde86469ca59d3393ee32bc13, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1679838884145 2023-03-26 13:54:45,756 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting abef16f9ca534e77b417bb184248502f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1679838884145 2023-03-26 13:54:45,757 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting b72fa16a9aa34577b4822f0b64020f33, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1679838885281 2023-03-26 13:54:45,757 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e499f404792240a5859dd19e5976f4c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1679838885281 2023-03-26 13:54:45,764 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#A#compaction#285 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:45,765 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#B#compaction#286 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:45,855 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:45,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:45,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:45,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:45,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:45,858 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:45,858 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:45,858 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:45,858 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:45,858 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,858 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,860 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,860 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:45,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:45,860 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,860 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:45,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:45,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:45,861 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:45,861 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:45,861 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:45,861 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:45,861 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:45,861 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:45,861 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:45,862 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-26 13:54:45,862 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:45,862 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:45,862 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:45,862 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:45,862 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:45,862 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:45,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:45,915 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:45,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:45,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34764 deadline: 1679838945923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:45,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838945923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:45,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:45,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34740 deadline: 1679838945924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:46,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838946026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:46,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34764 deadline: 1679838946026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:46,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34740 deadline: 1679838946027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,184 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/9a26994d134a40e49a04371da7328b9a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/9a26994d134a40e49a04371da7328b9a 2023-03-26 13:54:46,188 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a59be4541fa5742a2bd937206749175d/A of a59be4541fa5742a2bd937206749175d into 9a26994d134a40e49a04371da7328b9a(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:46,188 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:46,188 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/A, priority=13, startTime=1679838885754; duration=0sec 2023-03-26 13:54:46,188 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:46,188 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:46,190 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36818 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:46,190 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/C is initiating minor compaction (all files) 2023-03-26 13:54:46,190 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/C in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:46,190 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/42bb6fcc3f1e444883b8f85bddbead4a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/7af8ede8d1784c8a904e117af2dbcdcf, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/5458206b89154c32bdfdad871206592f] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=36.0 K 2023-03-26 13:54:46,191 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 42bb6fcc3f1e444883b8f85bddbead4a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1679838884113 2023-03-26 13:54:46,191 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 7af8ede8d1784c8a904e117af2dbcdcf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1679838884145 2023-03-26 13:54:46,191 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 5458206b89154c32bdfdad871206592f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1679838885281 2023-03-26 13:54:46,194 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/70f822758e734ff587695467db4a643f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/70f822758e734ff587695467db4a643f 2023-03-26 13:54:46,198 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a59be4541fa5742a2bd937206749175d/B of a59be4541fa5742a2bd937206749175d into 70f822758e734ff587695467db4a643f(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:46,198 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:46,198 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/B, priority=13, startTime=1679838885754; duration=0sec 2023-03-26 13:54:46,199 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:46,201 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#C#compaction#288 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:46,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:46,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838946229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:46,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34764 deadline: 1679838946229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:46,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34740 deadline: 1679838946229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,269 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/c54857700db3401c91ba24e94142f37c 2023-03-26 13:54:46,280 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/7887147a72774d52b81987cd81e53f41 2023-03-26 13:54:46,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:46,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838946530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:46,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34740 deadline: 1679838946531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:46,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34764 deadline: 1679838946533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,612 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/84429fa3df904ac1999a85eabbef1542 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/84429fa3df904ac1999a85eabbef1542 2023-03-26 13:54:46,617 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a59be4541fa5742a2bd937206749175d/C of a59be4541fa5742a2bd937206749175d into 84429fa3df904ac1999a85eabbef1542(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:46,617 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:46,617 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/C, priority=13, startTime=1679838885754; duration=0sec 2023-03-26 13:54:46,617 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:46,689 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/032b3cfa9de6444a88c70615cc24f86c 2023-03-26 13:54:46,694 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/c54857700db3401c91ba24e94142f37c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/c54857700db3401c91ba24e94142f37c 2023-03-26 13:54:46,698 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/c54857700db3401c91ba24e94142f37c, entries=150, sequenceid=306, filesize=11.9 K 2023-03-26 13:54:46,699 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/7887147a72774d52b81987cd81e53f41 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/7887147a72774d52b81987cd81e53f41 2023-03-26 13:54:46,704 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/7887147a72774d52b81987cd81e53f41, entries=150, sequenceid=306, filesize=11.9 K 2023-03-26 13:54:46,705 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/032b3cfa9de6444a88c70615cc24f86c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/032b3cfa9de6444a88c70615cc24f86c 2023-03-26 13:54:46,709 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/032b3cfa9de6444a88c70615cc24f86c, entries=150, sequenceid=306, filesize=11.9 K 2023-03-26 13:54:46,709 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a59be4541fa5742a2bd937206749175d in 847ms, sequenceid=306, compaction requested=false 2023-03-26 13:54:46,710 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:46,710 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:46,710 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:46,710 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:46,710 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,710 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:46,710 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:46,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,711 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:46,712 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:46,712 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:46,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:46,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6996b87c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:46,713 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:46,713 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:46,714 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:46,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:46,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:46,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:46,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:46,715 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,715 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:46,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:46,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:46,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:46,717 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:46,717 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:46,717 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:46,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:46,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:46,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:46,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:46,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:46,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:46,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:46,720 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:46,720 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:46,720 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:46,720 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:46,720 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:46,722 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:46,722 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:46,722 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:46,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:46,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:46,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:46,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:46,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:46,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:46,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:46,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:46,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:46,723 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:46,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:46,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:46,724 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:46,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:46,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,727 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,727 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:46,727 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:46,727 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:46,727 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:46,727 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:46,727 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,727 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:46,727 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:46,727 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:46,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:46,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:46,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:46,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:46,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:46,727 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:46,727 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:46,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:46,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:46,827 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:46,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:46,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:46,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:46,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:46,831 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:46,831 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:46,831 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:46,831 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:46,831 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:46,831 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:46,832 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:46,832 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:46,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:46,833 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:46,833 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:46,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:46,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:46,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:46,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:46,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:46,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:46,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:46,834 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:46,834 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:46,834 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:46,834 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:46,834 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:46,835 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:46,835 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:46,835 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 13:54:46,835 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:46,835 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:46,835 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:46,835 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:46,835 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:46,835 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:46,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:46,963 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:46,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838886975, max timestamp=1679838886995Num uniques 859; 2023-03-26 13:54:46,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:54:46,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838886975, max timestamp=1679838886995Num uniques 859; 2023-03-26 13:54:46,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838886975, max timestamp=1679838886995Num uniques 859; 2023-03-26 13:54:46,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:54:46,997 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:54:47,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:47,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838947021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:47,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:47,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838947022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:47,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:47,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34740 deadline: 1679838947036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:47,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:47,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838947036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:47,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:47,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34764 deadline: 1679838947040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:47,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:47,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838947125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:47,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:47,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838947125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:47,247 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/941e70d612a64c0bba6af57ebabca69c 2023-03-26 13:54:47,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:47,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838947327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:47,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:47,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838947328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:47,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:47,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838947629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:47,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:47,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838947631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:47,659 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/0ff466cf95dd4420a65f0940af9844ee 2023-03-26 13:54:48,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:48,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34740 deadline: 1679838948042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:48,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34764 deadline: 1679838948046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:48,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34804 deadline: 1679838948046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,069 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/1011f920f18c4ee8bf48ad21ab339be5 2023-03-26 13:54:48,073 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/941e70d612a64c0bba6af57ebabca69c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/941e70d612a64c0bba6af57ebabca69c 2023-03-26 13:54:48,077 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/941e70d612a64c0bba6af57ebabca69c, entries=150, sequenceid=321, filesize=11.9 K 2023-03-26 13:54:48,077 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/0ff466cf95dd4420a65f0940af9844ee as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/0ff466cf95dd4420a65f0940af9844ee 2023-03-26 13:54:48,080 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/0ff466cf95dd4420a65f0940af9844ee, entries=150, sequenceid=321, filesize=11.9 K 2023-03-26 13:54:48,081 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/1011f920f18c4ee8bf48ad21ab339be5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/1011f920f18c4ee8bf48ad21ab339be5 2023-03-26 13:54:48,084 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/1011f920f18c4ee8bf48ad21ab339be5, entries=150, sequenceid=321, filesize=11.9 K 2023-03-26 13:54:48,085 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for a59be4541fa5742a2bd937206749175d in 1250ms, sequenceid=321, compaction requested=true 2023-03-26 13:54:48,085 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:48,085 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:48,085 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:48,085 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:48,085 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,085 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:48,085 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:48,087 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,087 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,087 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,087 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a359ecd[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:48,089 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:48,089 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,090 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,090 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:48,090 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:48,090 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:48,090 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,090 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:48,091 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,091 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:48,091 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,091 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:48,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:48,094 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:48,094 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:48,094 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:48,094 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:48,094 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:48,097 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,097 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,097 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:48,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,097 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,099 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,105 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,105 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,105 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,105 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:48,105 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:48,105 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:48,106 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,106 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,106 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:48,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:48,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:48,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:48,106 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,106 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:48,106 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:48,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:48,138 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-26 13:54:48,139 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:48,139 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:48,139 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:48,139 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:48,139 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:48,139 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:48,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:48,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838948153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:48,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838948155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,206 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:48,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:48,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:48,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:48,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:48,210 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:48,210 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:48,210 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:48,210 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:48,210 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,210 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,212 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,212 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:48,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:48,212 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,212 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:48,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:48,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:48,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:48,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:48,214 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:48,214 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:48,214 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:48,214 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:48,214 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:48,214 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:48,214 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:48,214 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:48,214 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:48,215 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:48,215 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:48,215 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,215 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:48,215 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:48,216 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,216 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,217 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,217 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:48,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1da309ef[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:48,218 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:48,218 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,219 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,219 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:48,219 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:48,220 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:48,221 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,221 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:48,222 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:48,222 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,222 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,222 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:48,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:48,226 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:48,226 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:48,226 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:48,226 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:48,226 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:48,227 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,227 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,227 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:48,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:48,228 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,229 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,231 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:48,231 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:48,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:48,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:48,231 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,231 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:48,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:48,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:48,232 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:48,232 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:48,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:48,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:48,232 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,232 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,232 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,232 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,232 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,232 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:48,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838948258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,260 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:48,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838948258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,332 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:48,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:48,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:48,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:48,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:48,335 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:48,335 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:48,335 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:48,335 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:48,336 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,336 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,337 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:48,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:48,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:48,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:48,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:48,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:48,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:48,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:48,339 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:48,339 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:48,339 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:48,339 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:48,339 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:48,339 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:48,339 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:48,339 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:48,339 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:48,339 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:48,339 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:48,339 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,339 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:48,339 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:48,342 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,342 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,342 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,342 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:48,343 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:48,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ea5a36e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:48,343 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,344 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:48,345 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:48,345 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,345 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:48,346 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:48,346 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:48,346 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,346 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:48,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:48,349 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:48,349 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:48,349 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:48,349 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:48,349 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:48,350 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,351 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,351 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:48,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,351 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,352 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,354 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,354 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,355 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,355 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,355 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,355 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,355 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,355 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:48,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:48,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:48,355 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:48,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:48,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:48,355 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:48,355 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:48,355 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:48,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:48,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:48,455 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:48,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:48,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:48,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:48,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:48,458 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:48,458 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:48,458 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:48,458 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:48,459 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,459 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,461 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,461 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:48,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:48,461 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,461 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:48,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,462 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:48,462 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:48,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:48,462 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:48,463 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:48,463 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:48,463 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:48,463 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:48,463 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:48,463 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:48,463 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:48,463 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,463 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:48,463 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:48,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:48,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838948461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:48,464 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838948462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,464 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,465 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,465 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@322ca39e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:48,466 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:48,466 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,467 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,467 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:48,467 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,468 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:48,467 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:48,468 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:48,469 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,469 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:48,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:48,472 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:48,472 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:48,472 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:48,472 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:48,472 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:48,473 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,474 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:48,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,474 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,474 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,475 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:48,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,480 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:48,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:48,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:48,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:48,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:48,481 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,481 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,481 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,481 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,481 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:48,481 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:48,481 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:48,556 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/d00087c1999d43d0bda414c2a063ec7e 2023-03-26 13:54:48,566 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/c1575d18452f47c481210c1bbe7d0a89 2023-03-26 13:54:48,581 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:48,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:48,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:48,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:48,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:48,585 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:48,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:48,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:48,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:48,587 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,587 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,594 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,594 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:48,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:48,594 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,594 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:48,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:48,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:48,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:48,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:48,595 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:48,596 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:48,596 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:48,596 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:48,596 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,598 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,598 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,598 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:48,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14c980e0[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:48,599 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:48,599 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,601 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,601 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:48,601 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,601 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:48,601 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:48,601 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:48,602 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:48,602 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:48,602 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,602 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:48,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:48,604 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:48,605 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:48,605 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:48,605 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:48,605 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:48,607 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,607 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,607 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:48,608 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,609 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:48,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,612 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:48,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:48,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:48,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:48,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,612 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:48,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:48,612 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:48,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:48,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:48,712 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:48,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:48,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:48,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:48,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:48,715 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:48,715 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:48,716 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:48,716 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:48,716 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,716 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:48,717 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:48,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:48,717 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,718 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:48,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:48,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:48,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:48,719 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:48,719 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:48,719 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:48,719 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:48,719 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:48,721 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,721 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,721 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,721 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:48,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11a00fe7[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:48,723 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:48,723 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,725 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,725 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:48,725 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,725 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:48,725 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:48,725 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:48,727 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:48,727 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,727 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,727 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:48,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:48,729 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:48,729 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:48,729 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:48,729 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:48,729 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:48,731 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,731 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:48,731 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:48,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:48,731 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,733 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,734 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,735 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,735 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,735 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,735 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:48,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:48,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:48,735 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,735 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:48,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:48,735 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:48,735 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:48,735 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:48,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:48,735 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:48,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:48,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:48,735 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:48,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838948765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:48,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838948766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,835 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:48,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:48,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:48,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:48,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:48,838 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:48,838 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:48,838 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:48,838 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:48,839 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,839 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,841 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,841 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:48,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:48,841 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,841 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:48,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:48,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:48,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:48,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:48,842 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:48,842 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:48,842 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:48,842 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:48,842 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:48,842 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:48,842 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:48,842 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:48,843 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:48,843 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:48,843 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:48,843 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,843 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:48,843 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:48,844 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,844 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,844 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,844 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:48,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@525d606e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:48,846 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:48,846 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,847 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:48,847 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:48,847 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:48,847 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,847 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:48,848 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,848 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:48,848 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:48,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,848 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:48,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:48,850 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:48,850 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:48,850 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:48,850 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:48,850 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:48,853 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,853 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,853 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:48,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:48,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,855 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,856 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,858 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,858 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,858 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,858 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,858 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,858 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,858 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:48,859 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:48,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:48,859 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:48,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:48,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:48,859 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:48,859 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:48,859 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:48,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:48,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:48,959 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:48,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:48,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:48,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:48,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:48,963 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:48,963 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:48,963 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:48,963 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:48,963 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,963 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:48,964 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:48,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:48,965 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,965 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:48,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:48,966 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:48,966 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:48,966 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:48,966 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:48,966 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:48,966 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:48,966 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:48,966 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:48,966 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:48,966 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:48,966 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:48,966 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:48,966 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:48,966 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,966 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:48,966 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:48,971 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,971 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,971 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,971 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:48,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:48,973 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:48,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ba09e8[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:48,973 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,974 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,974 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:48,974 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,974 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:48,974 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:48,975 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:48,976 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,976 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:48,976 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,976 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:48,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:48,978 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:48,978 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:48,978 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:48,978 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:48,978 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:48,979 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,980 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:48,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,980 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:48,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:48,981 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,987 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,988 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=356 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/3b338a5c642d4bc0a12a07cd5fc81fb3 2023-03-26 13:54:48,990 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:48,991 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:48,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:48,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:48,990 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:48,991 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,991 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:48,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:48,991 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:48,991 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:48,991 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:48,991 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:48,991 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,991 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:48,991 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:48,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:48,991 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/d00087c1999d43d0bda414c2a063ec7e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/d00087c1999d43d0bda414c2a063ec7e 2023-03-26 13:54:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:48,995 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/d00087c1999d43d0bda414c2a063ec7e, entries=200, sequenceid=356, filesize=14.3 K 2023-03-26 13:54:48,996 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/c1575d18452f47c481210c1bbe7d0a89 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/c1575d18452f47c481210c1bbe7d0a89 2023-03-26 13:54:48,999 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/c1575d18452f47c481210c1bbe7d0a89, entries=150, sequenceid=356, filesize=11.9 K 2023-03-26 13:54:49,000 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/3b338a5c642d4bc0a12a07cd5fc81fb3 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/3b338a5c642d4bc0a12a07cd5fc81fb3 2023-03-26 13:54:49,003 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/3b338a5c642d4bc0a12a07cd5fc81fb3, entries=150, sequenceid=356, filesize=11.9 K 2023-03-26 13:54:49,004 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for a59be4541fa5742a2bd937206749175d in 866ms, sequenceid=356, compaction requested=true 2023-03-26 13:54:49,004 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:49,004 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:49,004 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:54:49,004 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:54:49,004 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:49,004 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:49,005 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51696 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:54:49,005 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/A is initiating minor compaction (all files) 2023-03-26 13:54:49,005 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/A in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:49,005 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/9a26994d134a40e49a04371da7328b9a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/c54857700db3401c91ba24e94142f37c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/941e70d612a64c0bba6af57ebabca69c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/d00087c1999d43d0bda414c2a063ec7e] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=50.5 K 2023-03-26 13:54:49,005 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:54:49,006 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/B is initiating minor compaction (all files) 2023-03-26 13:54:49,006 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/B in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:49,006 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/70f822758e734ff587695467db4a643f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/7887147a72774d52b81987cd81e53f41, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/0ff466cf95dd4420a65f0940af9844ee, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/c1575d18452f47c481210c1bbe7d0a89] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=48.1 K 2023-03-26 13:54:49,006 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 9a26994d134a40e49a04371da7328b9a, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1679838885281 2023-03-26 13:54:49,007 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting c54857700db3401c91ba24e94142f37c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1679838885293 2023-03-26 13:54:49,007 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 70f822758e734ff587695467db4a643f, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1679838885281 2023-03-26 13:54:49,007 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 941e70d612a64c0bba6af57ebabca69c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1679838885919 2023-03-26 13:54:49,007 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 7887147a72774d52b81987cd81e53f41, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1679838885293 2023-03-26 13:54:49,007 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting d00087c1999d43d0bda414c2a063ec7e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1679838887016 2023-03-26 13:54:49,008 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 0ff466cf95dd4420a65f0940af9844ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1679838885919 2023-03-26 13:54:49,008 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting c1575d18452f47c481210c1bbe7d0a89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1679838887016 2023-03-26 13:54:49,015 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#A#compaction#297 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:49,016 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#B#compaction#298 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:49,091 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:49,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:49,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:49,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:49,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:49,094 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:49,094 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:49,094 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:49,094 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:49,095 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,095 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:49,096 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,096 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:49,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:49,096 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,096 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:49,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:49,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:49,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:49,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:49,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:49,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:49,097 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:49,097 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:49,097 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:49,098 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:49,098 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:49,098 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:49,098 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:49,098 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 13:54:49,098 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:49,098 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:49,098 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:49,098 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:49,098 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:49,098 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:49,107 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/43e54a3ab648478dbfeef36dd5b7a344 2023-03-26 13:54:49,124 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/a008a43063114e9486f17a2caa177a71 2023-03-26 13:54:49,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:49,272 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:49,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838889304, max timestamp=1679838889337Num uniques 859; 2023-03-26 13:54:49,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838889304, max timestamp=1679838889337Num uniques 859; 2023-03-26 13:54:49,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:54:49,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838889304, max timestamp=1679838889337Num uniques 859; 2023-03-26 13:54:49,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:54:49,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:54:49,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:49,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838949379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:49,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838949382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,425 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/b42e0c3dcd094913a06cc69db544c2c8 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/b42e0c3dcd094913a06cc69db544c2c8 2023-03-26 13:54:49,432 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a59be4541fa5742a2bd937206749175d/B of a59be4541fa5742a2bd937206749175d into b42e0c3dcd094913a06cc69db544c2c8(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:49,432 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:49,432 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/B, priority=12, startTime=1679838889004; duration=0sec 2023-03-26 13:54:49,433 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:49,433 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:54:49,435 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/4ce1821371134761b4286cf204b55ac2 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/4ce1821371134761b4286cf204b55ac2 2023-03-26 13:54:49,436 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:54:49,436 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/C is initiating minor compaction (all files) 2023-03-26 13:54:49,436 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/C in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:49,436 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/84429fa3df904ac1999a85eabbef1542, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/032b3cfa9de6444a88c70615cc24f86c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/1011f920f18c4ee8bf48ad21ab339be5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/3b338a5c642d4bc0a12a07cd5fc81fb3] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=48.1 K 2023-03-26 13:54:49,437 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 84429fa3df904ac1999a85eabbef1542, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1679838885281 2023-03-26 13:54:49,437 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 032b3cfa9de6444a88c70615cc24f86c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1679838885293 2023-03-26 13:54:49,438 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 1011f920f18c4ee8bf48ad21ab339be5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1679838885919 2023-03-26 13:54:49,439 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 3b338a5c642d4bc0a12a07cd5fc81fb3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1679838887016 2023-03-26 13:54:49,440 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a59be4541fa5742a2bd937206749175d/A of a59be4541fa5742a2bd937206749175d into 4ce1821371134761b4286cf204b55ac2(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:49,441 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:49,441 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/A, priority=12, startTime=1679838889004; duration=0sec 2023-03-26 13:54:49,441 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:49,447 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#C#compaction#302 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:49,457 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/cd692c6c8d5d4927bd4ae05925a5b5ab as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/cd692c6c8d5d4927bd4ae05925a5b5ab 2023-03-26 13:54:49,462 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a59be4541fa5742a2bd937206749175d/C of a59be4541fa5742a2bd937206749175d into cd692c6c8d5d4927bd4ae05925a5b5ab(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:49,462 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:49,462 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/C, priority=12, startTime=1679838889004; duration=0sec 2023-03-26 13:54:49,462 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:49,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:49,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838949483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:49,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838949485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,542 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=367 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/28af2500a09e4ea5866d7cc132772130 2023-03-26 13:54:49,547 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/43e54a3ab648478dbfeef36dd5b7a344 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/43e54a3ab648478dbfeef36dd5b7a344 2023-03-26 13:54:49,551 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/43e54a3ab648478dbfeef36dd5b7a344, entries=150, sequenceid=367, filesize=11.9 K 2023-03-26 13:54:49,552 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/a008a43063114e9486f17a2caa177a71 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/a008a43063114e9486f17a2caa177a71 2023-03-26 13:54:49,556 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/a008a43063114e9486f17a2caa177a71, entries=150, sequenceid=367, filesize=11.9 K 2023-03-26 13:54:49,557 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/28af2500a09e4ea5866d7cc132772130 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/28af2500a09e4ea5866d7cc132772130 2023-03-26 13:54:49,561 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/28af2500a09e4ea5866d7cc132772130, entries=150, sequenceid=367, filesize=11.9 K 2023-03-26 13:54:49,562 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a59be4541fa5742a2bd937206749175d in 464ms, sequenceid=367, compaction requested=false 2023-03-26 13:54:49,562 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:49,562 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:49,562 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:49,562 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:49,562 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,562 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:49,562 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:49,564 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,564 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:49,564 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,564 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:49,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:49,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dac573f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:49,566 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:49,566 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,568 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:49,568 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:49,568 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:49,568 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,568 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:49,570 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:49,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:49,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:49,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:49,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:49,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:49,573 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:49,573 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:49,573 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:49,573 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:49,573 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:49,575 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,575 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,575 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:49,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,576 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,577 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:49,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:49,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:49,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:49,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:49,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:49,581 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:49,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:49,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:49,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:49,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:49,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:49,582 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:49,582 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:49,682 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:49,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:49,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:49,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:49,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:49,686 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:49,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:49,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:49,686 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-26 13:54:49,686 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:49,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:49,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:49,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:49,687 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:49,687 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:49,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:49,687 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:49,688 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,688 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:49,689 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:49,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:49,691 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,691 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:49,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:49,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:49,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:49,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:49,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:49,696 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:49,696 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:49,696 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:49,717 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:49,717 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:49,717 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:49,717 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:49,717 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:49,717 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:49,717 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:49,717 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:49,717 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,717 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:49,717 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:49,720 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:49,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:49,720 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:49,721 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,721 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:49,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:49,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:49,723 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:49,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c62b45a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:49,723 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:49,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838949719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,727 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:49,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34774 deadline: 1679838949725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,729 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,729 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:49,729 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:49,729 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:49,729 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,730 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:49,733 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:49,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:49,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:49,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:49,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:49,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,740 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:49,740 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:49,740 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:49,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:49,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:49,740 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:49,740 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:49,740 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:49,740 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:49,741 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:49,742 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,742 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:49,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,742 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:49,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:49,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:49,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:49,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:49,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,744 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,745 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,749 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:49,749 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:49,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:49,749 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,749 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:49,749 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:49,749 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,749 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,749 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:49,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:49,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:49,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:49,749 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,750 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,750 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:49,750 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:49,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:49,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:49,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838949825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:49,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838949828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,850 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:49,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:49,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:49,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:49,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:49,853 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:49,853 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:49,853 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:49,853 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:49,854 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,854 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:49,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:49,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:49,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:49,855 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,855 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:49,855 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:49,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:49,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:49,856 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:49,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:49,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:49,856 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:49,856 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:49,857 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:49,857 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:49,857 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:49,857 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:49,857 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:49,857 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:49,857 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:49,857 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:49,857 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:49,857 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,857 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:49,857 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:49,859 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,859 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:49,859 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,859 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:49,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:49,860 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:49,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7585f3a8[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:49,860 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,869 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,869 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:49,869 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:49,869 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,869 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:49,869 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:49,870 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:49,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:49,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:49,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:49,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:49,873 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:49,873 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:49,873 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:49,873 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:49,873 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:49,874 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,874 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:49,875 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:49,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,875 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:49,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,877 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:49,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,879 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,879 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:49,879 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:49,879 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,879 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:49,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:49,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:49,879 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:49,879 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:49,879 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,879 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,879 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:49,879 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:49,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:49,879 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:49,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:49,980 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:49,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:49,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:49,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:49,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:49,983 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:49,983 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:49,983 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:49,983 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:49,984 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,984 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:49,986 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:49,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:49,986 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:49,987 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,987 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:49,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:49,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:49,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:49,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:49,988 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:49,988 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:49,988 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:49,988 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:49,988 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:49,989 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:49,989 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,989 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:49,989 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:49,990 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,990 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:49,991 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,991 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:49,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:49,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:49,992 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@202551b9[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:49,992 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:49,992 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,994 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:49,994 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:49,994 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:49,994 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:49,994 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,994 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:49,995 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:49,995 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:49,995 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:49,995 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:49,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:49,998 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:49,999 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:49,999 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:49,999 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:49,999 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:50,002 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,002 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:50,002 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:50,002 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:50,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:50,003 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,006 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,006 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:50,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:50,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:50,006 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:50,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:50,006 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:50,006 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,007 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,007 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,007 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,007 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,007 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,007 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:50,007 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:50,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:50,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:50,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:50,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838950029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:50,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838950032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:50,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34740 deadline: 1679838950050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,054 DEBUG [Thread-1937] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4129 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., hostname=jenkins-hbase4.apache.org,42517,1679838814721, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:54:50,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:50,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34804 deadline: 1679838950052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,056 DEBUG [Thread-1943] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., hostname=jenkins-hbase4.apache.org,42517,1679838814721, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:54:50,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:50,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34764 deadline: 1679838950065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,067 DEBUG [Thread-1939] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., hostname=jenkins-hbase4.apache.org,42517,1679838814721, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:54:50,107 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:50,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:50,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:50,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:50,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:50,115 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:50,115 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:50,115 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:50,115 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:50,116 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,116 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,117 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:50,117 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:50,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,117 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,118 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:50,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:50,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:50,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:50,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:50,118 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:50,118 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:50,118 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:50,119 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:50,119 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:50,119 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:50,119 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:50,119 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:50,119 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:50,119 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:50,119 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:50,119 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,119 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:50,119 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:50,119 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/186b8f7d29a243e9b48f30a463544306 2023-03-26 13:54:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,121 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,121 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:50,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:50,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f0fff2d[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:50,124 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:50,124 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,126 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,127 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:50,127 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:50,127 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:50,127 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,127 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:50,128 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:50,128 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:50,128 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:50,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:50,128 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,131 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/af0d84c958c04c6097ad2eeec748c087 2023-03-26 13:54:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:50,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:50,132 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:50,132 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:50,132 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:50,132 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:50,132 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:50,134 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,134 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:50,134 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:50,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:50,135 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,137 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,140 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,140 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,140 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,140 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:50,140 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:50,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:50,140 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:50,140 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:50,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:50,140 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,141 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,141 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,141 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:50,141 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:50,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:50,241 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:50,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:50,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:50,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:50,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:50,244 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:50,244 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:50,244 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:50,244 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:50,244 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,244 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,247 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,247 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:50,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:50,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:50,247 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,247 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:50,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:50,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:50,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:50,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:50,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:50,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:50,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:50,249 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:50,249 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:50,249 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:50,249 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:50,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:50,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:50,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:50,249 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:50,250 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,250 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:50,251 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,251 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:50,252 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@469648f2[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:50,252 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,253 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,253 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:50,253 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:50,253 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:50,253 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,253 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:50,255 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,255 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:50,255 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:50,255 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:50,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:50,257 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:50,257 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:50,257 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:50,257 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:50,257 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:50,262 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:50,263 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,263 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:50,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:50,263 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,265 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:50,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,267 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,267 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,267 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,267 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,267 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:50,267 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:50,267 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:50,267 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:50,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:50,267 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:50,267 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,267 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:50,267 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:50,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:50,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:50,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:50,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34814 deadline: 1679838950332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:50,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838950336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,367 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:50,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:50,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:50,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:50,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:50,371 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:50,371 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:50,371 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:50,371 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:50,371 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,371 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,372 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:50,372 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:50,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:50,373 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,373 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:50,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:50,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:50,374 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:50,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:50,374 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:50,374 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:50,374 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:50,374 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:50,374 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:50,375 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,375 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:50,375 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:50,376 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:50,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:50,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:50,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:50,378 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:50,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c54fb3b[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:50,378 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,379 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,379 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:50,379 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:50,379 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:50,379 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,380 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:50,381 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,381 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:50,381 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:50,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:50,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:50,381 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:50,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:50,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:50,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:50,383 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:50,383 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:50,383 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:50,383 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:50,383 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:50,385 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,385 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,385 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:50,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:50,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:50,385 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,387 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:50,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,389 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,389 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:50,389 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:50,389 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:50,390 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,390 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,390 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,390 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,390 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:50,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:50,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:50,390 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:50,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:50,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:50,390 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:50,390 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:50,490 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:50,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:50,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:50,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:50,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:50,494 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:50,494 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:50,494 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:50,494 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:50,494 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,494 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,496 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,496 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:50,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:50,497 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,497 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:50,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:50,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:50,498 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:50,498 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:50,498 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:50,498 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:50,498 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:50,498 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:50,498 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:50,498 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:50,498 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:50,498 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:50,498 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:50,498 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,498 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:50,498 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:50,500 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,500 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:50,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:50,500 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,500 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:50,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:50,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:50,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:50,501 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:50,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1676c991[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:50,501 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,502 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,503 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:50,503 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:50,503 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:50,503 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,503 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:50,504 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,504 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:50,504 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:50,504 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:50,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:50,506 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:50,506 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:50,506 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:50,506 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:50,506 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:50,509 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,509 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,509 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:50,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:50,509 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:50,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:50,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,510 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:50,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,513 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,513 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,513 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,513 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,513 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:50,513 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:50,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:50,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:50,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:50,513 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,513 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:50,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:50,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:50,513 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:50,513 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:50,513 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:50,514 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:50,544 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/bdc4ec431bbe42cbb88f4bc668460266 2023-03-26 13:54:50,548 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/186b8f7d29a243e9b48f30a463544306 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/186b8f7d29a243e9b48f30a463544306 2023-03-26 13:54:50,551 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/186b8f7d29a243e9b48f30a463544306, entries=150, sequenceid=407, filesize=11.9 K 2023-03-26 13:54:50,551 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/af0d84c958c04c6097ad2eeec748c087 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/af0d84c958c04c6097ad2eeec748c087 2023-03-26 13:54:50,555 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/af0d84c958c04c6097ad2eeec748c087, entries=150, sequenceid=407, filesize=11.9 K 2023-03-26 13:54:50,555 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/bdc4ec431bbe42cbb88f4bc668460266 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/bdc4ec431bbe42cbb88f4bc668460266 2023-03-26 13:54:50,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/bdc4ec431bbe42cbb88f4bc668460266, entries=150, sequenceid=407, filesize=11.9 K 2023-03-26 13:54:50,559 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for a59be4541fa5742a2bd937206749175d in 873ms, sequenceid=407, compaction requested=true 2023-03-26 13:54:50,559 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:50,559 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:50,559 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:50,559 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:50,559 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:50,559 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:50,560 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:50,560 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:50,560 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/A is initiating minor compaction (all files) 2023-03-26 13:54:50,560 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/B is initiating minor compaction (all files) 2023-03-26 13:54:50,560 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/A in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:50,560 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/B in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:50,560 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/4ce1821371134761b4286cf204b55ac2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/43e54a3ab648478dbfeef36dd5b7a344, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/186b8f7d29a243e9b48f30a463544306] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=36.3 K 2023-03-26 13:54:50,560 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/b42e0c3dcd094913a06cc69db544c2c8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/a008a43063114e9486f17a2caa177a71, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/af0d84c958c04c6097ad2eeec748c087] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=36.3 K 2023-03-26 13:54:50,561 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting b42e0c3dcd094913a06cc69db544c2c8, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1679838887016 2023-03-26 13:54:50,561 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 4ce1821371134761b4286cf204b55ac2, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1679838887016 2023-03-26 13:54:50,561 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting a008a43063114e9486f17a2caa177a71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679838888149 2023-03-26 13:54:50,561 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 43e54a3ab648478dbfeef36dd5b7a344, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679838888149 2023-03-26 13:54:50,561 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting af0d84c958c04c6097ad2eeec748c087, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1679838889359 2023-03-26 13:54:50,561 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 186b8f7d29a243e9b48f30a463544306, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1679838889359 2023-03-26 13:54:50,568 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#B#compaction#306 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:50,570 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#A#compaction#307 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:50,581 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/4428f115bd0f41c8934ac4ecb30c07b9 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/4428f115bd0f41c8934ac4ecb30c07b9 2023-03-26 13:54:50,581 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/48db7690205b4b9785a778b7ef190168 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/48db7690205b4b9785a778b7ef190168 2023-03-26 13:54:50,585 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a59be4541fa5742a2bd937206749175d/B of a59be4541fa5742a2bd937206749175d into 4428f115bd0f41c8934ac4ecb30c07b9(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:50,585 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:50,585 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/B, priority=13, startTime=1679838890559; duration=0sec 2023-03-26 13:54:50,585 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:50,585 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:50,585 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a59be4541fa5742a2bd937206749175d/A of a59be4541fa5742a2bd937206749175d into 48db7690205b4b9785a778b7ef190168(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:50,585 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:50,586 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/A, priority=13, startTime=1679838890559; duration=0sec 2023-03-26 13:54:50,586 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:50,587 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:50,587 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/C is initiating minor compaction (all files) 2023-03-26 13:54:50,587 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/C in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:50,587 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/cd692c6c8d5d4927bd4ae05925a5b5ab, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/28af2500a09e4ea5866d7cc132772130, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/bdc4ec431bbe42cbb88f4bc668460266] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=36.3 K 2023-03-26 13:54:50,587 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting cd692c6c8d5d4927bd4ae05925a5b5ab, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=356, earliestPutTs=1679838887016 2023-03-26 13:54:50,587 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 28af2500a09e4ea5866d7cc132772130, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=367, earliestPutTs=1679838888149 2023-03-26 13:54:50,588 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting bdc4ec431bbe42cbb88f4bc668460266, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1679838889359 2023-03-26 13:54:50,593 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#C#compaction#308 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:50,601 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/23d809eaabbb4d1d91dd3a7cc1f4adbd as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/23d809eaabbb4d1d91dd3a7cc1f4adbd 2023-03-26 13:54:50,604 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a59be4541fa5742a2bd937206749175d/C of a59be4541fa5742a2bd937206749175d into 23d809eaabbb4d1d91dd3a7cc1f4adbd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:50,604 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:50,604 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/C, priority=13, startTime=1679838890559; duration=0sec 2023-03-26 13:54:50,604 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:50,614 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:50,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:50,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:50,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:50,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:50,616 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:50,617 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:50,617 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:50,617 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:50,617 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,617 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,618 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:50,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:50,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:50,618 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,619 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,619 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:50,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:50,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:50,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:50,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:50,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:50,620 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:50,620 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:50,620 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:50,620 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:50,620 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:50,620 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:50,620 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:50,620 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 13:54:50,620 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:50,621 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:50,621 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:50,621 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:50,621 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:50,621 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:50,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:50,839 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:50,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838890844, max timestamp=1679838890865Num uniques 859; 2023-03-26 13:54:50,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:54:50,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838890844, max timestamp=1679838890865Num uniques 859; 2023-03-26 13:54:50,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:54:50,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838890844, max timestamp=1679838890865Num uniques 859; 2023-03-26 13:54:50,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:54:50,891 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:50,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838950889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:50,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838950890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:50,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838950993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:50,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:50,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838950994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,029 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/c07c5608e3ae47e696035375c6f2d6ee 2023-03-26 13:54:51,041 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/7c77dc0f5a344ceba06aef807b8db355 2023-03-26 13:54:51,050 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=421 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/1d499ff9941a4e179e08e415244ae56d 2023-03-26 13:54:51,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/c07c5608e3ae47e696035375c6f2d6ee as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/c07c5608e3ae47e696035375c6f2d6ee 2023-03-26 13:54:51,057 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/c07c5608e3ae47e696035375c6f2d6ee, entries=150, sequenceid=421, filesize=11.9 K 2023-03-26 13:54:51,057 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/7c77dc0f5a344ceba06aef807b8db355 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/7c77dc0f5a344ceba06aef807b8db355 2023-03-26 13:54:51,060 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/7c77dc0f5a344ceba06aef807b8db355, entries=150, sequenceid=421, filesize=11.9 K 2023-03-26 13:54:51,061 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/1d499ff9941a4e179e08e415244ae56d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/1d499ff9941a4e179e08e415244ae56d 2023-03-26 13:54:51,064 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/1d499ff9941a4e179e08e415244ae56d, entries=150, sequenceid=421, filesize=11.9 K 2023-03-26 13:54:51,064 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a59be4541fa5742a2bd937206749175d in 444ms, sequenceid=421, compaction requested=false 2023-03-26 13:54:51,064 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:51,064 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:51,065 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:51,065 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:51,065 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,065 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:51,065 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:51,066 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,066 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,066 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,067 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:51,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7660bae5[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:51,069 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:51,069 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,071 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:51,072 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:51,073 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:51,073 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:51,073 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:51,073 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,073 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:51,073 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:51,073 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:51,073 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:51,073 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:51,073 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:51,073 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:51,075 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,075 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,075 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:51,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:51,078 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,079 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:51,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:51,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:51,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:51,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,085 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:51,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:51,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:51,085 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:51,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,085 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:51,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:51,185 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:51,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:51,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:51,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:51,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:51,189 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:51,189 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:51,189 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:51,189 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:51,189 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,189 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,190 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:51,190 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:51,191 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,191 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:51,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:51,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:51,192 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:51,192 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:51,192 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:51,192 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:51,192 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:51,192 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:51,192 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:51,192 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-26 13:54:51,193 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:51,193 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:51,193 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:51,193 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:51,193 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:51,193 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:51,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:51,198 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:51,203 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/ecf2dbd0c6a746eba9f3c1a46aac951a 2023-03-26 13:54:51,215 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/f0b83157744141e18c714b0ee1aa1590 2023-03-26 13:54:51,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:51,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838951213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,226 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:51,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838951224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,232 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/ce1f8056976945f59f5bc04eab72a60a 2023-03-26 13:54:51,239 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/ecf2dbd0c6a746eba9f3c1a46aac951a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/ecf2dbd0c6a746eba9f3c1a46aac951a 2023-03-26 13:54:51,242 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/ecf2dbd0c6a746eba9f3c1a46aac951a, entries=150, sequenceid=456, filesize=11.9 K 2023-03-26 13:54:51,243 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/f0b83157744141e18c714b0ee1aa1590 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/f0b83157744141e18c714b0ee1aa1590 2023-03-26 13:54:51,247 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/f0b83157744141e18c714b0ee1aa1590, entries=150, sequenceid=456, filesize=11.9 K 2023-03-26 13:54:51,247 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/ce1f8056976945f59f5bc04eab72a60a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/ce1f8056976945f59f5bc04eab72a60a 2023-03-26 13:54:51,251 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/ce1f8056976945f59f5bc04eab72a60a, entries=150, sequenceid=456, filesize=11.9 K 2023-03-26 13:54:51,251 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for a59be4541fa5742a2bd937206749175d in 59ms, sequenceid=456, compaction requested=true 2023-03-26 13:54:51,251 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:51,252 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:51,252 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:51,252 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:51,252 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,252 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:51,252 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:51,253 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,253 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,254 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,254 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:51,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:51,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c5bd03[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:51,255 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:51,255 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,256 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,256 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:51,256 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:51,256 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:51,256 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,256 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:51,258 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,258 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:51,259 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,259 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:51,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:51,262 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:51,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:51,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:51,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:51,262 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:51,264 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,264 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:51,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,264 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:51,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,265 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,266 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,270 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,270 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,270 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,270 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,270 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,270 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,270 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,270 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:51,270 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:51,270 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:51,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:51,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:51,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:51,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:51,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:51,271 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:51,271 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:51,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:51,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:51,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:51,321 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 13:54:51,322 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:51,322 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:51,322 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:51,322 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:51,322 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:51,322 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:51,351 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:51,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838951348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:51,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838951351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,371 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:51,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:51,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:51,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:51,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:51,374 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:51,374 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:51,374 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:51,374 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:51,374 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,374 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,375 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:51,375 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:51,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,376 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,376 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:51,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:51,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:51,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:51,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:51,377 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:51,377 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:51,377 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:51,377 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:51,377 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:51,379 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,379 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,380 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,380 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:51,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39f31343[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:51,381 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:51,381 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,382 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,382 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:51,382 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:51,382 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:51,382 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,382 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:51,383 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:51,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,383 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:51,384 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:51,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:51,386 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:51,386 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:51,386 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:51,386 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:51,386 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:51,387 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,387 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,387 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,387 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,389 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,391 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,391 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:51,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:51,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:51,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:51,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:51,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:51,391 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:51,391 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,392 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:51,392 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:51,392 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:51,392 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:51,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:51,392 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,392 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,392 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:51,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838951452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:51,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838951454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,492 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:51,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:51,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:51,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:51,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:51,495 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:51,495 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:51,495 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:51,495 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:51,496 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,496 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,497 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:51,497 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:51,497 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,498 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:51,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:51,499 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:51,499 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:51,499 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:51,499 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:51,499 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:51,499 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:51,508 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,508 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,508 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,508 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:51,510 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:51,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19e97fad[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:51,510 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,511 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,511 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:51,511 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,511 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:51,511 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:51,511 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:51,514 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,514 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:51,514 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:51,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,514 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:51,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:51,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:51,516 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:51,516 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:51,516 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:51,516 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:51,516 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:51,518 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,518 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,518 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:51,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:51,519 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,520 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,525 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,525 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,525 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,525 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:51,525 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,525 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:51,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:51,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:51,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:51,526 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:51,525 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,526 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:51,526 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,526 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:51,526 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:51,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:51,626 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:51,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:51,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:51,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:51,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:51,629 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:51,629 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:51,629 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:51,629 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:51,630 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,630 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,634 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,634 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:51,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:51,634 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,634 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:51,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:51,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:51,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:51,636 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:51,637 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:51,637 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:51,637 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:51,637 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:51,637 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:51,637 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:51,637 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:51,637 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:51,637 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,637 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:51,637 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:51,638 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,638 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,639 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,639 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:51,640 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:51,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a2d49e7[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:51,640 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,642 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,642 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:51,642 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:51,642 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:51,642 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,642 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:51,646 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,646 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:51,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:51,648 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:51,648 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:51,649 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:51,649 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:51,649 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:51,651 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,651 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,651 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:51,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,651 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,653 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,655 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:51,655 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:51,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:51,655 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:51,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:51,655 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,655 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,655 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,655 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:51,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:51,655 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:51,655 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:51,655 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,655 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:51,655 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:51,656 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:51,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838951655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,658 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:51,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838951657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,730 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/6efc3c2746eb4e139ed4d98fa4e34a89 2023-03-26 13:54:51,747 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/0f0253c6676945929a1fceb571eb9830 2023-03-26 13:54:51,756 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:51,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:51,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:51,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:51,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:51,759 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:51,759 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:51,759 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:51,759 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:51,760 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,760 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,761 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,761 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:51,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:51,761 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,761 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,762 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:51,762 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:51,762 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:51,763 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:51,763 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:51,763 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:51,763 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:51,763 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:51,766 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,766 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,766 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,766 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:51,767 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:51,767 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@452fdc04[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:51,768 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:51,768 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,769 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:51,769 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,769 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:51,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:51,772 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:51,772 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:51,772 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:51,772 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:51,772 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:51,774 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,774 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,774 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,774 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,776 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:51,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:51,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:51,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:51,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:51,780 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:51,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:51,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:51,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:51,780 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:51,781 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:51,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,781 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,781 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,781 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,881 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:51,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:51,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:51,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:51,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:51,884 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:51,884 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:51,884 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:51,884 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:51,885 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,885 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:51,886 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:51,886 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,886 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:51,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:51,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:51,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:51,888 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:51,888 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:51,888 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:51,888 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:51,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:51,890 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,890 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,891 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,891 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:51,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6467a6e9[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:51,893 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:51,893 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,894 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:51,894 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,894 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:51,897 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,897 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:51,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:51,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:51,900 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:51,900 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:51,900 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:51,900 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:51,900 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:51,901 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,901 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,901 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:51,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:51,902 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:51,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,903 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,905 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:51,906 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:51,906 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:51,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:51,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:51,906 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:51,906 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,906 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:51,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:51,906 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:51,906 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,906 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:51,906 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,906 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:51,906 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:51,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:51,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:51,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838951958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:51,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:51,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838951959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,006 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:52,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:52,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:52,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:52,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:52,012 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:52,013 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:52,013 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:52,013 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:52,013 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,013 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:52,014 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:52,014 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:52,015 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,015 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:52,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:52,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:52,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:52,016 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:52,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:52,016 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:52,016 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:52,016 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:52,016 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:52,017 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:52,017 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:52,017 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:52,017 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:52,018 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,018 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:52,018 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:52,020 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:52,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:52,021 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:52,021 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:52,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:52,022 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36a1b9df[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:52,022 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:52,022 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:52,023 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:52,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:52,023 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:52,024 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:52,024 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,024 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:52,024 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:52,025 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,025 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:52,025 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:52,025 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:52,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:52,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:52,028 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:52,028 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:52,028 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:52,028 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:52,028 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:52,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:52,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:52,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:52,030 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:52,031 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:52,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:52,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,034 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,034 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:52,034 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:52,034 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:52,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:52,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:52,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:52,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:52,034 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,034 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:52,034 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:52,035 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:52,035 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:52,035 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,035 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:52,035 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:52,035 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:52,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:52,136 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:52,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:52,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:52,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:52,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:52,139 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:52,140 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:52,140 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:52,140 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:52,140 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,140 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:52,141 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,142 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:52,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:52,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:52,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:52,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:52,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:52,143 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:52,143 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:52,143 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:52,143 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:52,143 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:52,143 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:52,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:52,145 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,145 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:52,145 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:52,145 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:52,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:52,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:52,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3643ac9d[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:52,146 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:52,146 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:52,147 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:52,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:52,147 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:52,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:52,148 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,148 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:52,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:52,150 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:52,150 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:52,150 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,150 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:52,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:52,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:52,153 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:52,153 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:52,153 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:52,153 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:52,153 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:52,154 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,154 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,154 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:52,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:52,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:52,155 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:52,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:52,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:52,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:52,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:52,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:52,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:52,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:52,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:52,158 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:52,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:52,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:52,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:52,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:52,158 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:52,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:52,159 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:52,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:52,160 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=468 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/487af930246f45ba95d4ce23da53f8d4 2023-03-26 13:54:52,165 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/6efc3c2746eb4e139ed4d98fa4e34a89 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/6efc3c2746eb4e139ed4d98fa4e34a89 2023-03-26 13:54:52,168 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/6efc3c2746eb4e139ed4d98fa4e34a89, entries=150, sequenceid=468, filesize=11.9 K 2023-03-26 13:54:52,169 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/0f0253c6676945929a1fceb571eb9830 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/0f0253c6676945929a1fceb571eb9830 2023-03-26 13:54:52,172 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/0f0253c6676945929a1fceb571eb9830, entries=150, sequenceid=468, filesize=11.9 K 2023-03-26 13:54:52,174 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/487af930246f45ba95d4ce23da53f8d4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/487af930246f45ba95d4ce23da53f8d4 2023-03-26 13:54:52,177 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/487af930246f45ba95d4ce23da53f8d4, entries=150, sequenceid=468, filesize=11.9 K 2023-03-26 13:54:52,178 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a59be4541fa5742a2bd937206749175d in 857ms, sequenceid=468, compaction requested=true 2023-03-26 13:54:52,178 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:52,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:52,178 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:54:52,178 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:54:52,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:52,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:52,179 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:54:52,179 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/A is initiating minor compaction (all files) 2023-03-26 13:54:52,179 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/A in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:52,179 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/48db7690205b4b9785a778b7ef190168, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/c07c5608e3ae47e696035375c6f2d6ee, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/ecf2dbd0c6a746eba9f3c1a46aac951a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/6efc3c2746eb4e139ed4d98fa4e34a89] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=48.3 K 2023-03-26 13:54:52,180 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 48db7690205b4b9785a778b7ef190168, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1679838889359 2023-03-26 13:54:52,180 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:54:52,180 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/B is initiating minor compaction (all files) 2023-03-26 13:54:52,180 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/B in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:52,180 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting c07c5608e3ae47e696035375c6f2d6ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1679838889698 2023-03-26 13:54:52,180 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/4428f115bd0f41c8934ac4ecb30c07b9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/7c77dc0f5a344ceba06aef807b8db355, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/f0b83157744141e18c714b0ee1aa1590, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/0f0253c6676945929a1fceb571eb9830] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=48.3 K 2023-03-26 13:54:52,180 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting ecf2dbd0c6a746eba9f3c1a46aac951a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1679838890881 2023-03-26 13:54:52,181 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 4428f115bd0f41c8934ac4ecb30c07b9, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1679838889359 2023-03-26 13:54:52,181 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 6efc3c2746eb4e139ed4d98fa4e34a89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1679838891206 2023-03-26 13:54:52,181 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 7c77dc0f5a344ceba06aef807b8db355, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1679838889698 2023-03-26 13:54:52,181 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting f0b83157744141e18c714b0ee1aa1590, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1679838890881 2023-03-26 13:54:52,182 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 0f0253c6676945929a1fceb571eb9830, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1679838891206 2023-03-26 13:54:52,188 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#A#compaction#318 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:52,189 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#B#compaction#319 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:52,201 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/aa87a7f8231744daa374cee577c33646 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/aa87a7f8231744daa374cee577c33646 2023-03-26 13:54:52,205 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a59be4541fa5742a2bd937206749175d/A of a59be4541fa5742a2bd937206749175d into aa87a7f8231744daa374cee577c33646(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:52,205 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:52,205 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/A, priority=12, startTime=1679838892178; duration=0sec 2023-03-26 13:54:52,205 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:52,205 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:54:52,206 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:54:52,206 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/C is initiating minor compaction (all files) 2023-03-26 13:54:52,206 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/C in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:52,207 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/23d809eaabbb4d1d91dd3a7cc1f4adbd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/1d499ff9941a4e179e08e415244ae56d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/ce1f8056976945f59f5bc04eab72a60a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/487af930246f45ba95d4ce23da53f8d4] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=48.3 K 2023-03-26 13:54:52,207 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 23d809eaabbb4d1d91dd3a7cc1f4adbd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1679838889359 2023-03-26 13:54:52,207 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 1d499ff9941a4e179e08e415244ae56d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=421, earliestPutTs=1679838889698 2023-03-26 13:54:52,207 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting ce1f8056976945f59f5bc04eab72a60a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1679838890881 2023-03-26 13:54:52,208 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 487af930246f45ba95d4ce23da53f8d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1679838891206 2023-03-26 13:54:52,214 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#C#compaction#320 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:52,259 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:52,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:52,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:52,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:52,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:52,262 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:52,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:52,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:52,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:52,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:52,263 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:52,263 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:52,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:52,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:52,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:52,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:52,266 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:52,266 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:52,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:52,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:52,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:52,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:52,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:52,266 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:52,266 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:52,266 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-26 13:54:52,267 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:52,267 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:52,267 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:52,267 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:52,267 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:52,267 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:52,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:52,463 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:52,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:52,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:52,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838952480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838952480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:52,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838952585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:52,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838952585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,602 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/6f879f9ad97e49f9ae724909a83e8323 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6f879f9ad97e49f9ae724909a83e8323 2023-03-26 13:54:52,606 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a59be4541fa5742a2bd937206749175d/B of a59be4541fa5742a2bd937206749175d into 6f879f9ad97e49f9ae724909a83e8323(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:52,606 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:52,606 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/B, priority=12, startTime=1679838892178; duration=0sec 2023-03-26 13:54:52,606 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:52,622 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/115957e9af88432882ee2cac516e39b4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/115957e9af88432882ee2cac516e39b4 2023-03-26 13:54:52,626 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a59be4541fa5742a2bd937206749175d/C of a59be4541fa5742a2bd937206749175d into 115957e9af88432882ee2cac516e39b4(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:52,626 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:52,626 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/C, priority=12, startTime=1679838892178; duration=0sec 2023-03-26 13:54:52,627 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:52,677 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/f76eef3a4d974255af173cb87086709b 2023-03-26 13:54:52,787 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:52,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838952787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:52,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:52,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838952788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,089 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/e483e3d32d4d45f2a08020fb8596bb35 2023-03-26 13:54:53,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:53,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838953089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,093 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:53,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838953091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,503 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/76e57f6e3f1f42b68df62bbbb4d026f8 2023-03-26 13:54:53,510 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/f76eef3a4d974255af173cb87086709b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/f76eef3a4d974255af173cb87086709b 2023-03-26 13:54:53,515 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/f76eef3a4d974255af173cb87086709b, entries=150, sequenceid=489, filesize=11.9 K 2023-03-26 13:54:53,516 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/e483e3d32d4d45f2a08020fb8596bb35 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/e483e3d32d4d45f2a08020fb8596bb35 2023-03-26 13:54:53,520 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/e483e3d32d4d45f2a08020fb8596bb35, entries=150, sequenceid=489, filesize=11.9 K 2023-03-26 13:54:53,520 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/76e57f6e3f1f42b68df62bbbb4d026f8 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/76e57f6e3f1f42b68df62bbbb4d026f8 2023-03-26 13:54:53,524 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/76e57f6e3f1f42b68df62bbbb4d026f8, entries=150, sequenceid=489, filesize=11.9 K 2023-03-26 13:54:53,525 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for a59be4541fa5742a2bd937206749175d in 1259ms, sequenceid=489, compaction requested=false 2023-03-26 13:54:53,525 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:53,525 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:53,525 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:53,525 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:53,525 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,525 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:53,525 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:53,527 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:53,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:53,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:53,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d3789ad[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:53,529 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:53,529 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,530 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,531 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:53,531 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:53,531 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:53,531 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,531 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:53,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:53,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:53,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:53,533 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:53,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:53,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:53,536 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:53,536 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:53,536 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:53,536 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:53,536 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:53,539 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,539 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:53,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,539 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:53,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:53,539 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:53,541 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,543 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,543 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:53,543 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:53,543 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,543 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:53,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:53,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:53,543 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,544 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,543 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:53,543 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:53,544 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:53,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:53,544 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:53,544 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:53,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:53,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:53,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:53,597 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-26 13:54:53,597 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:53,597 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:53,598 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:53,598 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:53,598 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:53,598 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:53,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:53,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838953631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:53,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838953632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,644 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:53,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:53,647 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/5977fadd436d409694407398aae9fa0a 2023-03-26 13:54:53,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:53,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:53,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:53,649 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:53,649 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:53,649 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:53,649 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:53,649 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,649 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:53,650 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,650 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:53,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:53,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:53,651 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:53,651 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:53,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:53,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:53,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:53,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:53,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:53,661 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:53,673 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:53,673 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:53,673 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:53,673 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:53,673 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:53,674 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:53,674 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:53,674 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:53,674 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,684 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:53,684 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:53,685 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:53,685 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:53,686 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,686 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:53,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:53,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54be360c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:53,688 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:53,688 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,695 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:53,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:53,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:53,695 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,695 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:53,697 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:53,697 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:53,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:53,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:53,699 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:53,699 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:53,699 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:53,699 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:53,699 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:53,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:53,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:53,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:53,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,702 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:53,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,704 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,706 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:53,706 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:53,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:53,707 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:53,707 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:53,707 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:53,706 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:53,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:53,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:53,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:53,707 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:53,707 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:53,708 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:53,708 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,708 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,708 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,708 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,736 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/a33199196eec484a888e16a5ea74f552 2023-03-26 13:54:53,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:53,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838953736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:53,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838953736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,807 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:53,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:53,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:53,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:53,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:53,810 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:53,810 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:53,810 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:53,810 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:53,811 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,811 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:53,813 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:53,813 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:53,813 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,814 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:53,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:53,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:53,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:53,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:53,815 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:53,815 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:53,815 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:53,815 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:53,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:53,817 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,817 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:53,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:53,819 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:53,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fea0205[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:53,819 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,821 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,821 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:53,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:53,821 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:53,821 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:53,823 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,823 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:53,823 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:53,823 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:53,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:53,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:53,825 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:53,825 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:53,825 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:53,825 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:53,825 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:53,829 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,829 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:53,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,832 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:53,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:53,833 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,835 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,838 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,838 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:53,839 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:53,838 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:53,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:53,839 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:53,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:53,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:53,839 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:53,839 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,839 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,839 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,839 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,839 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:53,839 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:53,839 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:53,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:53,939 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:53,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:53,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:53,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:53,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838953939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:53,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838953939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:53,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:53,942 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:53,942 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:53,943 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:53,943 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:53,943 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,943 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:53,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:53,944 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:53,945 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,945 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:53,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:53,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:53,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:53,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:53,946 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:53,946 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:53,946 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:53,946 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:53,946 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:53,948 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,948 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:53,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:53,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44667c21[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:53,949 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:53,949 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,954 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,955 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:53,955 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:53,955 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:53,955 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,955 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:53,956 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,956 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:53,956 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:53,956 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:53,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:53,958 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:53,958 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:53,958 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:53,958 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:53,958 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:53,960 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,960 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,960 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:53,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:53,960 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,962 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:53,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:53,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:53,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:53,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:53,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:53,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:53,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:53,964 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:53,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:53,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:53,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:53,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:53,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:53,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:53,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:53,965 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:53,965 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:54,065 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:54,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:54,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:54,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:54,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:54,068 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:54,068 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:54,068 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:54,068 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:54,069 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,069 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:54,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:54,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34764 deadline: 1679838954069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,071 DEBUG [Thread-1939] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., hostname=jenkins-hbase4.apache.org,42517,1679838814721, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:54:54,071 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:54,071 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:54,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:54,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:54,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:54,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:54,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:54,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:54,073 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:54,073 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:54,073 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:54,073 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:54,073 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:54,073 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:54,073 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:54,073 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:54,073 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:54,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:54,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:54,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:54,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:54,075 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,075 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:54,075 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:54,075 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:54,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:54,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:54,077 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:54,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75adefe1[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:54,077 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:54,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:54,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:54,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:54,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:54,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:54,078 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,078 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:54,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:54,082 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:54,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34804 deadline: 1679838954081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,083 DEBUG [Thread-1943] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., hostname=jenkins-hbase4.apache.org,42517,1679838814721, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:54:54,084 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,084 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:54,084 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,084 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:54,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:54,087 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:54,087 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:54,087 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:54,087 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:54,087 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:54,088 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,088 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:54,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:54,088 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:54,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:54,089 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:54,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:54,090 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,092 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,092 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:54,092 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:54,092 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,092 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:54,092 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:54,092 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:54,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:54,092 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:54,092 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:54,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:54,092 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:54,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:54,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:54,092 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:54,093 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:54,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:54,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34740 deadline: 1679838954092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,094 DEBUG [Thread-1937] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., hostname=jenkins-hbase4.apache.org,42517,1679838814721, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:54:54,163 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/5bca65691396427a92a18ab83a4cd596 2023-03-26 13:54:54,167 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/5977fadd436d409694407398aae9fa0a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/5977fadd436d409694407398aae9fa0a 2023-03-26 13:54:54,170 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/5977fadd436d409694407398aae9fa0a, entries=150, sequenceid=505, filesize=11.9 K 2023-03-26 13:54:54,171 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/a33199196eec484a888e16a5ea74f552 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/a33199196eec484a888e16a5ea74f552 2023-03-26 13:54:54,174 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/a33199196eec484a888e16a5ea74f552, entries=150, sequenceid=505, filesize=11.9 K 2023-03-26 13:54:54,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/5bca65691396427a92a18ab83a4cd596 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/5bca65691396427a92a18ab83a4cd596 2023-03-26 13:54:54,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/5bca65691396427a92a18ab83a4cd596, entries=150, sequenceid=505, filesize=11.9 K 2023-03-26 13:54:54,179 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for a59be4541fa5742a2bd937206749175d in 582ms, sequenceid=505, compaction requested=true 2023-03-26 13:54:54,179 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:54,179 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:54,179 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:54,179 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:54,179 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:54,179 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:54,180 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:54,180 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/A is initiating minor compaction (all files) 2023-03-26 13:54:54,180 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/A in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:54,180 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/aa87a7f8231744daa374cee577c33646, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/f76eef3a4d974255af173cb87086709b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/5977fadd436d409694407398aae9fa0a] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=36.6 K 2023-03-26 13:54:54,181 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:54,181 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting aa87a7f8231744daa374cee577c33646, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1679838891206 2023-03-26 13:54:54,181 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/B is initiating minor compaction (all files) 2023-03-26 13:54:54,181 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/B in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:54,181 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6f879f9ad97e49f9ae724909a83e8323, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/e483e3d32d4d45f2a08020fb8596bb35, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/a33199196eec484a888e16a5ea74f552] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=36.6 K 2023-03-26 13:54:54,181 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting f76eef3a4d974255af173cb87086709b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1679838891346 2023-03-26 13:54:54,181 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 6f879f9ad97e49f9ae724909a83e8323, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1679838891206 2023-03-26 13:54:54,182 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 5977fadd436d409694407398aae9fa0a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1679838892475 2023-03-26 13:54:54,182 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e483e3d32d4d45f2a08020fb8596bb35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1679838891346 2023-03-26 13:54:54,182 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting a33199196eec484a888e16a5ea74f552, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1679838892475 2023-03-26 13:54:54,188 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#A#compaction#327 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:54,190 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#B#compaction#328 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:54,193 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:54,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:54,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:54,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:54,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:54,200 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:54,200 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:54,200 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:54,200 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:54,201 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,201 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:54,202 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:54,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:54,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:54,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:54,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:54,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:54,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:54,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:54,205 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:54,205 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:54,205 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:54,205 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:54,205 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:54,205 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:54,205 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:54,205 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:54,205 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:54,205 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-26 13:54:54,205 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:54,206 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:54,206 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:54,206 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:54,206 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:54,206 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:54,207 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/5fb614a57799449db539e9f8c05af0bc as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/5fb614a57799449db539e9f8c05af0bc 2023-03-26 13:54:54,211 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a59be4541fa5742a2bd937206749175d/B of a59be4541fa5742a2bd937206749175d into 5fb614a57799449db539e9f8c05af0bc(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:54,211 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:54,211 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/B, priority=13, startTime=1679838894179; duration=0sec 2023-03-26 13:54:54,211 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:54:54,212 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:54:54,212 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:54:54,212 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/C is initiating minor compaction (all files) 2023-03-26 13:54:54,213 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/C in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:54,213 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/115957e9af88432882ee2cac516e39b4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/76e57f6e3f1f42b68df62bbbb4d026f8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/5bca65691396427a92a18ab83a4cd596] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=36.6 K 2023-03-26 13:54:54,213 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 115957e9af88432882ee2cac516e39b4, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=468, earliestPutTs=1679838891206 2023-03-26 13:54:54,213 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 76e57f6e3f1f42b68df62bbbb4d026f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1679838891346 2023-03-26 13:54:54,216 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 5bca65691396427a92a18ab83a4cd596, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1679838892475 2023-03-26 13:54:54,216 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/aa5210527c4c4d21a57dcc011534dd94 2023-03-26 13:54:54,221 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#C#compaction#330 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:54:54,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:54,246 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:54,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:54,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838954275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:54,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838954277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,380 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:54,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838954378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:54,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838954379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:54,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838954582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:54,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838954582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,600 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/829d66411492458fbee654c15917cb2a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/829d66411492458fbee654c15917cb2a 2023-03-26 13:54:54,604 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a59be4541fa5742a2bd937206749175d/A of a59be4541fa5742a2bd937206749175d into 829d66411492458fbee654c15917cb2a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:54,605 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:54,605 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/A, priority=13, startTime=1679838894179; duration=0sec 2023-03-26 13:54:54,605 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:54,630 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/56bc42c7a691446ca3d8d19a6bf83a44 2023-03-26 13:54:54,640 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/0805b8749f794b5da2b9f6d2ca516b34 2023-03-26 13:54:54,644 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/aa5210527c4c4d21a57dcc011534dd94 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/aa5210527c4c4d21a57dcc011534dd94 2023-03-26 13:54:54,657 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/5772f12effba4cb493170910b9ebd7f2 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/5772f12effba4cb493170910b9ebd7f2 2023-03-26 13:54:54,659 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/aa5210527c4c4d21a57dcc011534dd94, entries=150, sequenceid=523, filesize=11.9 K 2023-03-26 13:54:54,660 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/56bc42c7a691446ca3d8d19a6bf83a44 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/56bc42c7a691446ca3d8d19a6bf83a44 2023-03-26 13:54:54,665 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a59be4541fa5742a2bd937206749175d/C of a59be4541fa5742a2bd937206749175d into 5772f12effba4cb493170910b9ebd7f2(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:54:54,665 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:54,665 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/C, priority=13, startTime=1679838894179; duration=0sec 2023-03-26 13:54:54,665 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:54:54,673 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/56bc42c7a691446ca3d8d19a6bf83a44, entries=150, sequenceid=523, filesize=11.9 K 2023-03-26 13:54:54,674 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/0805b8749f794b5da2b9f6d2ca516b34 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/0805b8749f794b5da2b9f6d2ca516b34 2023-03-26 13:54:54,678 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/0805b8749f794b5da2b9f6d2ca516b34, entries=150, sequenceid=523, filesize=11.9 K 2023-03-26 13:54:54,678 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for a59be4541fa5742a2bd937206749175d in 473ms, sequenceid=523, compaction requested=false 2023-03-26 13:54:54,678 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:54,678 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:54,679 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:54,679 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:54,679 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:54,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:54,682 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:54,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:54,683 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:54,683 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:54,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:54,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5086a90e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:54,685 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:54,685 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:54,686 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:54,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:54,686 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:54,686 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:54,686 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:54,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:54,687 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,687 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:54,687 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:54,688 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:54,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:54,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:54,691 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:54,691 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:54,691 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:54,691 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:54,691 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:54,693 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,693 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:54,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,693 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:54,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,694 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:54,696 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,700 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,700 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:54,700 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:54,700 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:54,701 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:54,700 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:54,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:54,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:54,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:54,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:54,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:54,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:54,701 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:54,701 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:54,801 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:54,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:54,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:54,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:54,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:54,804 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:54,805 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:54,805 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:54,805 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:54,805 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,805 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:54,806 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:54,806 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:54,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:54,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:54,807 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,807 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:54,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:54,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:54,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:54,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:54,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:54,808 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:54,808 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:54,808 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:54,808 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:54,808 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:54,808 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:54,808 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:54,808 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-26 13:54:54,809 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:54,809 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:54,809 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:54,809 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:54,809 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:54,809 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:54,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:54,889 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:54,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:54,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:54,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838954922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:54,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838954922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:55,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:55,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838955025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:55,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:55,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838955025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:55,215 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/3919b03f597247989f7cd2251b22e10f 2023-03-26 13:54:55,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:55,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838955228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:55,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:55,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838955228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:55,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:55,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838955531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:55,534 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:55,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838955532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:55,625 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/388dc639619740dea02f0cbc21dcb9b6 2023-03-26 13:54:56,035 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/204f163af093460198b70c28833d32f1 2023-03-26 13:54:56,039 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/3919b03f597247989f7cd2251b22e10f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/3919b03f597247989f7cd2251b22e10f 2023-03-26 13:54:56,040 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:56,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838956037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:56,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:56,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838956039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:56,043 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/3919b03f597247989f7cd2251b22e10f, entries=150, sequenceid=540, filesize=11.9 K 2023-03-26 13:54:56,043 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/388dc639619740dea02f0cbc21dcb9b6 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/388dc639619740dea02f0cbc21dcb9b6 2023-03-26 13:54:56,049 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/388dc639619740dea02f0cbc21dcb9b6, entries=150, sequenceid=540, filesize=11.9 K 2023-03-26 13:54:56,051 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/204f163af093460198b70c28833d32f1 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/204f163af093460198b70c28833d32f1 2023-03-26 13:54:56,055 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/204f163af093460198b70c28833d32f1, entries=150, sequenceid=540, filesize=11.9 K 2023-03-26 13:54:56,056 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for a59be4541fa5742a2bd937206749175d in 1248ms, sequenceid=540, compaction requested=true 2023-03-26 13:54:56,056 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:56,056 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:56,056 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:56,056 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:56,056 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:56,062 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:56,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:56,063 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:56,063 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:56,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fc36331[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:56,065 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:56,065 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:56,066 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:56,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:56,066 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:56,066 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:56,066 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:56,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:56,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:56,068 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:56,068 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:56,069 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:56,069 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:56,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:56,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:56,072 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:56,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:56,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:56,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:56,072 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:56,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:56,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:56,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:56,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:56,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:56,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:56,075 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:56,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:56,076 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:56,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:56,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:56,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:56,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:56,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:56,079 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:56,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:56,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:56,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:56,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:56,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:56,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:56,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:56,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:56,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:56,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:56,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:56,080 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:56,080 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:56,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:56,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:56,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:56,180 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:56,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:56,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:56,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:56,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:56,183 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:56,183 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:56,183 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:56,183 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:56,184 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:56,184 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:56,185 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:56,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:56,185 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:56,185 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:56,185 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:56,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:56,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:56,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:56,186 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:56,186 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:56,186 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:56,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:56,187 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:56,187 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:56,187 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-26 13:54:56,187 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:56,187 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:56,187 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:56,187 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:56,187 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:56,187 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:56,598 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/1a86daa1f73240928b2e9c7274c17364 2023-03-26 13:54:57,008 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/9fb797e8b9f64bf1a49743a0caa34924 2023-03-26 13:54:57,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:57,046 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:57,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:57,060 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:57,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838957058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838957059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:57,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838957162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:57,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838957162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:57,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838957366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:57,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838957366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,419 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=559 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/c4f2fd4065c7467a9ca945974d060384 2023-03-26 13:54:57,422 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/1a86daa1f73240928b2e9c7274c17364 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/1a86daa1f73240928b2e9c7274c17364 2023-03-26 13:54:57,426 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/1a86daa1f73240928b2e9c7274c17364, entries=150, sequenceid=559, filesize=11.9 K 2023-03-26 13:54:57,426 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/9fb797e8b9f64bf1a49743a0caa34924 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/9fb797e8b9f64bf1a49743a0caa34924 2023-03-26 13:54:57,429 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/9fb797e8b9f64bf1a49743a0caa34924, entries=150, sequenceid=559, filesize=11.9 K 2023-03-26 13:54:57,430 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/c4f2fd4065c7467a9ca945974d060384 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/c4f2fd4065c7467a9ca945974d060384 2023-03-26 13:54:57,433 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/c4f2fd4065c7467a9ca945974d060384, entries=150, sequenceid=559, filesize=11.9 K 2023-03-26 13:54:57,434 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for a59be4541fa5742a2bd937206749175d in 1247ms, sequenceid=559, compaction requested=true 2023-03-26 13:54:57,434 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:57,434 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:57,434 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:57,434 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:57,434 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,434 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:57,434 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:57,436 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,436 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:57,437 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:57,437 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:57,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:57,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18339d9c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:57,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:57,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:57,439 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:57,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:57,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:57,439 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:57,439 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:57,439 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,439 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:57,439 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:57,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:57,441 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:57,441 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:57,441 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:57,444 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:57,444 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:57,444 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:57,444 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:57,445 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:57,446 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:57,446 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:57,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:57,446 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:57,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:57,447 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:57,448 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,454 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,454 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:57,454 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:57,454 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,454 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:57,454 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:57,454 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:57,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:57,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:57,454 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:57,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:57,454 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:57,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:57,454 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:57,454 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:57,454 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:57,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:57,555 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:57,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:57,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:54:57,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:54:57,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:54:57,558 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:54:57,558 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:57,558 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:54:57,558 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:54:57,558 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:57,558 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:57,559 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:57,559 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:57,560 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,560 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:54:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:57,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:54:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:57,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:54:57,561 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:54:57,561 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:54:57,561 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:54:57,561 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:54:57,561 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:54:57,561 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:54:57,561 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:54:57,561 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:57,561 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. started... 2023-03-26 13:54:57,561 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-26 13:54:57,562 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:57,562 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:57,562 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:57,562 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:57,562 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:57,562 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:54:57,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:57,672 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. as already flushing 2023-03-26 13:54:57,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:57,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838957706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:57,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838957707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,813 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:57,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838957811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:57,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838957811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:57,973 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/b1f98ee81e864eb1abc477d99cfb27bc 2023-03-26 13:54:58,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:58,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838958015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,016 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:58,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838958015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,313 DEBUG [Thread-1946] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x270a3c0c to 127.0.0.1:54332 2023-03-26 13:54:58,313 DEBUG [Thread-1946] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:58,313 DEBUG [Thread-1950] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06bbe763 to 127.0.0.1:54332 2023-03-26 13:54:58,313 DEBUG [Thread-1950] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:58,314 DEBUG [Thread-1952] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c8736d4 to 127.0.0.1:54332 2023-03-26 13:54:58,314 DEBUG [Thread-1952] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:58,315 DEBUG [Thread-1954] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x451c882f to 127.0.0.1:54332 2023-03-26 13:54:58,315 DEBUG [Thread-1954] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:58,316 DEBUG [Thread-1948] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1e067bfc to 127.0.0.1:54332 2023-03-26 13:54:58,316 DEBUG [Thread-1948] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:58,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:58,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:54:58,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34774 deadline: 1679838958317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34814 deadline: 1679838958317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,385 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/8371a179f1c84987ba6e55fbfbe91ca5 2023-03-26 13:54:58,794 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/df93af02dc4641b6bb52a7bd9f2451b1 2023-03-26 13:54:58,797 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/b1f98ee81e864eb1abc477d99cfb27bc as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/b1f98ee81e864eb1abc477d99cfb27bc 2023-03-26 13:54:58,801 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/b1f98ee81e864eb1abc477d99cfb27bc, entries=100, sequenceid=572, filesize=9.5 K 2023-03-26 13:54:58,802 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/8371a179f1c84987ba6e55fbfbe91ca5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/8371a179f1c84987ba6e55fbfbe91ca5 2023-03-26 13:54:58,806 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/8371a179f1c84987ba6e55fbfbe91ca5, entries=100, sequenceid=572, filesize=9.5 K 2023-03-26 13:54:58,807 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/df93af02dc4641b6bb52a7bd9f2451b1 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/df93af02dc4641b6bb52a7bd9f2451b1 2023-03-26 13:54:58,810 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/df93af02dc4641b6bb52a7bd9f2451b1, entries=100, sequenceid=572, filesize=9.5 K 2023-03-26 13:54:58,811 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for a59be4541fa5742a2bd937206749175d in 1249ms, sequenceid=572, compaction requested=true 2023-03-26 13:54:58,811 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:54:58,811 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:54:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:54:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:54:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:54:58,811 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:54:58,813 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,813 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:58,813 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:58,813 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:54:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:58,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:54:58,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27bab9c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:54:58,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:54:58,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:58,816 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:58,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:58,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:54:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:54:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:54:58,817 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,817 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:54:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:54:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:58,819 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:54:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:54:58,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:54:58,821 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:54:58,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:54:58,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:54:58,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:54:58,821 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:54:58,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:58,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:54:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:54:58,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:58,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:54:58,823 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:58,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:58,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a59be4541fa5742a2bd937206749175d 2023-03-26 13:54:58,823 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-26 13:54:58,823 DEBUG [Thread-1935] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1b40c526 to 127.0.0.1:54332 2023-03-26 13:54:58,823 DEBUG [Thread-1935] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:58,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838897694, max timestamp=1679838898822Num uniques 859; 2023-03-26 13:54:58,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838897694, max timestamp=1679838898822Num uniques 859; 2023-03-26 13:54:58,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:54:58,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:58,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:54:58,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838897694, max timestamp=1679838898822Num uniques 859; 2023-03-26 13:54:58,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:58,824 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:54:58,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:54:58,825 DEBUG [Thread-1941] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c6d5986 to 127.0.0.1:54332 2023-03-26 13:54:58,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:54:58,825 DEBUG [Thread-1941] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:54:58,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:58,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:54:58,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:58,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:54:58,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:54:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:58,826 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:54:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:54:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,829 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:54:58,829 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:54:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:54:58,829 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:54:58,829 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:54:58,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:54:58,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:54:58,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:54:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:54:58,829 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:58,830 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:54:58,830 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:54:58,830 DEBUG [Thread-1945] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:54:58,830 DEBUG [Thread-1945] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:54:58,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:54:58,830 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:58,830 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:54:58,830 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:54:58,930 DEBUG [Thread-1945] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:54:58,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:54:59,234 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/144018325dc147149a5e6f75a7ca19ed 2023-03-26 13:54:59,645 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/cc833ee2363b4e30b69b917e29ef73fa 2023-03-26 13:55:00,053 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=594 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/f1610ef0301f4a1bae70d4aa581f82a1 2023-03-26 13:55:00,057 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/144018325dc147149a5e6f75a7ca19ed as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/144018325dc147149a5e6f75a7ca19ed 2023-03-26 13:55:00,059 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/144018325dc147149a5e6f75a7ca19ed, entries=150, sequenceid=594, filesize=11.9 K 2023-03-26 13:55:00,060 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/cc833ee2363b4e30b69b917e29ef73fa as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/cc833ee2363b4e30b69b917e29ef73fa 2023-03-26 13:55:00,063 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/cc833ee2363b4e30b69b917e29ef73fa, entries=150, sequenceid=594, filesize=11.9 K 2023-03-26 13:55:00,063 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/f1610ef0301f4a1bae70d4aa581f82a1 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/f1610ef0301f4a1bae70d4aa581f82a1 2023-03-26 13:55:00,066 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/f1610ef0301f4a1bae70d4aa581f82a1, entries=150, sequenceid=594, filesize=11.9 K 2023-03-26 13:55:00,067 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=0 B/0 for a59be4541fa5742a2bd937206749175d in 1244ms, sequenceid=594, compaction requested=true 2023-03-26 13:55:00,067 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:55:00,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:00,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:00,067 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 13:55:00,067 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 13:55:00,067 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:00,069 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71660 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 13:55:00,069 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71660 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 13:55:00,069 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/A is initiating minor compaction (all files) 2023-03-26 13:55:00,069 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/B is initiating minor compaction (all files) 2023-03-26 13:55:00,069 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/A in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:55:00,069 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/B in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:55:00,069 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/829d66411492458fbee654c15917cb2a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/aa5210527c4c4d21a57dcc011534dd94, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/3919b03f597247989f7cd2251b22e10f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/1a86daa1f73240928b2e9c7274c17364, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/b1f98ee81e864eb1abc477d99cfb27bc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/144018325dc147149a5e6f75a7ca19ed] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=70.0 K 2023-03-26 13:55:00,069 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/5fb614a57799449db539e9f8c05af0bc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/56bc42c7a691446ca3d8d19a6bf83a44, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/388dc639619740dea02f0cbc21dcb9b6, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/9fb797e8b9f64bf1a49743a0caa34924, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/8371a179f1c84987ba6e55fbfbe91ca5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/cc833ee2363b4e30b69b917e29ef73fa] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=70.0 K 2023-03-26 13:55:00,069 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 829d66411492458fbee654c15917cb2a, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1679838892475 2023-03-26 13:55:00,069 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 5fb614a57799449db539e9f8c05af0bc, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1679838892475 2023-03-26 13:55:00,069 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting aa5210527c4c4d21a57dcc011534dd94, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1679838893625 2023-03-26 13:55:00,070 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 56bc42c7a691446ca3d8d19a6bf83a44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1679838893625 2023-03-26 13:55:00,070 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 3919b03f597247989f7cd2251b22e10f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1679838894248 2023-03-26 13:55:00,070 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 388dc639619740dea02f0cbc21dcb9b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1679838894248 2023-03-26 13:55:00,070 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 1a86daa1f73240928b2e9c7274c17364, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1679838894899 2023-03-26 13:55:00,070 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 9fb797e8b9f64bf1a49743a0caa34924, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1679838894899 2023-03-26 13:55:00,070 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting b1f98ee81e864eb1abc477d99cfb27bc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1679838897057 2023-03-26 13:55:00,070 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 8371a179f1c84987ba6e55fbfbe91ca5, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1679838897057 2023-03-26 13:55:00,070 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 144018325dc147149a5e6f75a7ca19ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1679838897694 2023-03-26 13:55:00,071 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting cc833ee2363b4e30b69b917e29ef73fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1679838897694 2023-03-26 13:55:00,080 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#A#compaction#345 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:00,081 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#B#compaction#346 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:00,488 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/f1c4b3e7d9554b78bf06df9084c30c55 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/f1c4b3e7d9554b78bf06df9084c30c55 2023-03-26 13:55:00,489 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/2a2fd500acf84bc5b5fe1b5fda81df8f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/2a2fd500acf84bc5b5fe1b5fda81df8f 2023-03-26 13:55:00,492 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a59be4541fa5742a2bd937206749175d/A of a59be4541fa5742a2bd937206749175d into f1c4b3e7d9554b78bf06df9084c30c55(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:00,492 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:55:00,492 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/A, priority=10, startTime=1679838900067; duration=0sec 2023-03-26 13:55:00,492 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:00,492 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 13:55:00,492 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a59be4541fa5742a2bd937206749175d/B of a59be4541fa5742a2bd937206749175d into 2a2fd500acf84bc5b5fe1b5fda81df8f(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:00,492 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:55:00,493 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/B, priority=10, startTime=1679838900067; duration=0sec 2023-03-26 13:55:00,493 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:00,493 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71660 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 13:55:00,494 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a59be4541fa5742a2bd937206749175d/C is initiating minor compaction (all files) 2023-03-26 13:55:00,494 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a59be4541fa5742a2bd937206749175d/C in TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:55:00,494 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/5772f12effba4cb493170910b9ebd7f2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/0805b8749f794b5da2b9f6d2ca516b34, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/204f163af093460198b70c28833d32f1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/c4f2fd4065c7467a9ca945974d060384, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/df93af02dc4641b6bb52a7bd9f2451b1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/f1610ef0301f4a1bae70d4aa581f82a1] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp, totalSize=70.0 K 2023-03-26 13:55:00,494 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 5772f12effba4cb493170910b9ebd7f2, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1679838892475 2023-03-26 13:55:00,494 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 0805b8749f794b5da2b9f6d2ca516b34, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1679838893625 2023-03-26 13:55:00,494 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 204f163af093460198b70c28833d32f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1679838894248 2023-03-26 13:55:00,495 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting c4f2fd4065c7467a9ca945974d060384, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=559, earliestPutTs=1679838894899 2023-03-26 13:55:00,495 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting df93af02dc4641b6bb52a7bd9f2451b1, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1679838897057 2023-03-26 13:55:00,495 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting f1610ef0301f4a1bae70d4aa581f82a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=594, earliestPutTs=1679838897694 2023-03-26 13:55:00,502 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a59be4541fa5742a2bd937206749175d#C#compaction#347 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:00,910 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/c01b76e6330446f98eb5886f2c06dbcd as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/c01b76e6330446f98eb5886f2c06dbcd 2023-03-26 13:55:00,913 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in a59be4541fa5742a2bd937206749175d/C of a59be4541fa5742a2bd937206749175d into c01b76e6330446f98eb5886f2c06dbcd(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:00,913 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:55:00,913 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d., storeName=a59be4541fa5742a2bd937206749175d/C, priority=10, startTime=1679838900067; duration=0sec 2023-03-26 13:55:00,913 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:04,086 DEBUG [Thread-1939] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x025522dc to 127.0.0.1:54332 2023-03-26 13:55:04,086 DEBUG [Thread-1939] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:55:04,103 DEBUG [Thread-1937] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x47ae15dc to 127.0.0.1:54332 2023-03-26 13:55:04,103 DEBUG [Thread-1937] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:55:04,104 DEBUG [Thread-1943] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x31182447 to 127.0.0.1:54332 2023-03-26 13:55:04,104 DEBUG [Thread-1943] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:55:04,104 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-26 13:55:04,104 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 139 2023-03-26 13:55:04,104 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 74 2023-03-26 13:55:04,104 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 73 2023-03-26 13:55:04,104 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 136 2023-03-26 13:55:04,104 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 67 2023-03-26 13:55:04,104 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-26 13:55:04,104 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(398): read 5610 2023-03-26 13:55:04,104 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(398): read 5530 2023-03-26 13:55:04,104 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(398): read 5695 2023-03-26 13:55:04,104 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(398): read 5622 2023-03-26 13:55:04,104 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(398): read 5552 2023-03-26 13:55:04,104 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-26 13:55:04,104 INFO [Listener at localhost/41531] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-26 13:55:04,104 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5b4a16d4 to 127.0.0.1:54332 2023-03-26 13:55:04,105 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:55:04,105 INFO [Listener at localhost/41531] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-26 13:55:04,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-26 13:55:04,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-26 13:55:04,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-26 13:55:04,109 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838904108"}]},"ts":"1679838904108"} 2023-03-26 13:55:04,109 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-26 13:55:04,111 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-26 13:55:04,111 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a59be4541fa5742a2bd937206749175d, UNASSIGN}] 2023-03-26 13:55:04,112 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a59be4541fa5742a2bd937206749175d, UNASSIGN 2023-03-26 13:55:04,113 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=a59be4541fa5742a2bd937206749175d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:04,113 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838904113"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838904113"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838904113"}]},"ts":"1679838904113"} 2023-03-26 13:55:04,114 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:55:04,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-26 13:55:04,265 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a59be4541fa5742a2bd937206749175d 2023-03-26 13:55:04,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a59be4541fa5742a2bd937206749175d, disabling compactions & flushes 2023-03-26 13:55:04,266 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:55:04,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:55:04,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. after waiting 0 ms 2023-03-26 13:55:04,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:55:04,266 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing a59be4541fa5742a2bd937206749175d 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-26 13:55:04,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=A 2023-03-26 13:55:04,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:04,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=B 2023-03-26 13:55:04,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:04,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a59be4541fa5742a2bd937206749175d, store=C 2023-03-26 13:55:04,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:04,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-26 13:55:04,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/e9d50c6593b84236bd6cb16e23574f75 2023-03-26 13:55:04,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-26 13:55:05,081 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/10a34a1e0342450e997f23a1d72cbf87 2023-03-26 13:55:05,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-26 13:55:05,489 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/a0684935a10c4689b8cfec54179f4821 2023-03-26 13:55:05,493 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/A/e9d50c6593b84236bd6cb16e23574f75 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/e9d50c6593b84236bd6cb16e23574f75 2023-03-26 13:55:05,496 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/e9d50c6593b84236bd6cb16e23574f75, entries=150, sequenceid=603, filesize=11.9 K 2023-03-26 13:55:05,496 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/B/10a34a1e0342450e997f23a1d72cbf87 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/10a34a1e0342450e997f23a1d72cbf87 2023-03-26 13:55:05,499 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/10a34a1e0342450e997f23a1d72cbf87, entries=150, sequenceid=603, filesize=11.9 K 2023-03-26 13:55:05,500 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/.tmp/C/a0684935a10c4689b8cfec54179f4821 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/a0684935a10c4689b8cfec54179f4821 2023-03-26 13:55:05,502 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/a0684935a10c4689b8cfec54179f4821, entries=150, sequenceid=603, filesize=11.9 K 2023-03-26 13:55:05,503 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for a59be4541fa5742a2bd937206749175d in 1237ms, sequenceid=603, compaction requested=false 2023-03-26 13:55:05,523 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/32c04a2eb6594ac29a6f943e6dcaed5b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/499ba14fe3de436584235367cdf437eb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/f9871748f3fa4aad927edf382a2f9a63, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/477d8465992a4b1a8afda585987b4026, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/062ad768f1474ad8a12bccfede06f58d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/6418062110c74fbebba1da143fc12c77, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/a1e8265bd1284776850e90676a7dddb8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/73553a6392dd408da2dea16852d5f631, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/8d7063a83c104366aa6b18c2adeffaac, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/e8aeda416fc1402a9be17ce377a98e2c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/6ae2f04015ae46a29cc980c34dda9662, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/121e11a209894c069ffc83f2c863a1c8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/0a5f7858a28c4da98845722d913309df, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/e2ec04acde86469ca59d3393ee32bc13, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/9a26994d134a40e49a04371da7328b9a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/b72fa16a9aa34577b4822f0b64020f33, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/c54857700db3401c91ba24e94142f37c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/941e70d612a64c0bba6af57ebabca69c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/d00087c1999d43d0bda414c2a063ec7e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/4ce1821371134761b4286cf204b55ac2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/43e54a3ab648478dbfeef36dd5b7a344, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/48db7690205b4b9785a778b7ef190168, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/186b8f7d29a243e9b48f30a463544306, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/c07c5608e3ae47e696035375c6f2d6ee, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/ecf2dbd0c6a746eba9f3c1a46aac951a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/aa87a7f8231744daa374cee577c33646, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/6efc3c2746eb4e139ed4d98fa4e34a89, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/f76eef3a4d974255af173cb87086709b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/829d66411492458fbee654c15917cb2a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/5977fadd436d409694407398aae9fa0a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/aa5210527c4c4d21a57dcc011534dd94, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/3919b03f597247989f7cd2251b22e10f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/1a86daa1f73240928b2e9c7274c17364, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/b1f98ee81e864eb1abc477d99cfb27bc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/144018325dc147149a5e6f75a7ca19ed] to archive 2023-03-26 13:55:05,524 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 13:55:05,525 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/32c04a2eb6594ac29a6f943e6dcaed5b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/32c04a2eb6594ac29a6f943e6dcaed5b 2023-03-26 13:55:05,526 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/499ba14fe3de436584235367cdf437eb to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/499ba14fe3de436584235367cdf437eb 2023-03-26 13:55:05,527 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/f9871748f3fa4aad927edf382a2f9a63 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/f9871748f3fa4aad927edf382a2f9a63 2023-03-26 13:55:05,528 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/477d8465992a4b1a8afda585987b4026 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/477d8465992a4b1a8afda585987b4026 2023-03-26 13:55:05,529 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/062ad768f1474ad8a12bccfede06f58d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/062ad768f1474ad8a12bccfede06f58d 2023-03-26 13:55:05,529 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/6418062110c74fbebba1da143fc12c77 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/6418062110c74fbebba1da143fc12c77 2023-03-26 13:55:05,530 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/a1e8265bd1284776850e90676a7dddb8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/a1e8265bd1284776850e90676a7dddb8 2023-03-26 13:55:05,531 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/73553a6392dd408da2dea16852d5f631 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/73553a6392dd408da2dea16852d5f631 2023-03-26 13:55:05,532 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/8d7063a83c104366aa6b18c2adeffaac to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/8d7063a83c104366aa6b18c2adeffaac 2023-03-26 13:55:05,532 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/e8aeda416fc1402a9be17ce377a98e2c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/e8aeda416fc1402a9be17ce377a98e2c 2023-03-26 13:55:05,533 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/6ae2f04015ae46a29cc980c34dda9662 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/6ae2f04015ae46a29cc980c34dda9662 2023-03-26 13:55:05,534 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/121e11a209894c069ffc83f2c863a1c8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/121e11a209894c069ffc83f2c863a1c8 2023-03-26 13:55:05,534 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/0a5f7858a28c4da98845722d913309df to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/0a5f7858a28c4da98845722d913309df 2023-03-26 13:55:05,535 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/e2ec04acde86469ca59d3393ee32bc13 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/e2ec04acde86469ca59d3393ee32bc13 2023-03-26 13:55:05,536 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/9a26994d134a40e49a04371da7328b9a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/9a26994d134a40e49a04371da7328b9a 2023-03-26 13:55:05,537 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/b72fa16a9aa34577b4822f0b64020f33 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/b72fa16a9aa34577b4822f0b64020f33 2023-03-26 13:55:05,537 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/c54857700db3401c91ba24e94142f37c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/c54857700db3401c91ba24e94142f37c 2023-03-26 13:55:05,538 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/941e70d612a64c0bba6af57ebabca69c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/941e70d612a64c0bba6af57ebabca69c 2023-03-26 13:55:05,539 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/d00087c1999d43d0bda414c2a063ec7e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/d00087c1999d43d0bda414c2a063ec7e 2023-03-26 13:55:05,539 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/4ce1821371134761b4286cf204b55ac2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/4ce1821371134761b4286cf204b55ac2 2023-03-26 13:55:05,540 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/43e54a3ab648478dbfeef36dd5b7a344 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/43e54a3ab648478dbfeef36dd5b7a344 2023-03-26 13:55:05,541 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/48db7690205b4b9785a778b7ef190168 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/48db7690205b4b9785a778b7ef190168 2023-03-26 13:55:05,541 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/186b8f7d29a243e9b48f30a463544306 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/186b8f7d29a243e9b48f30a463544306 2023-03-26 13:55:05,542 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/c07c5608e3ae47e696035375c6f2d6ee to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/c07c5608e3ae47e696035375c6f2d6ee 2023-03-26 13:55:05,543 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/ecf2dbd0c6a746eba9f3c1a46aac951a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/ecf2dbd0c6a746eba9f3c1a46aac951a 2023-03-26 13:55:05,543 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/aa87a7f8231744daa374cee577c33646 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/aa87a7f8231744daa374cee577c33646 2023-03-26 13:55:05,544 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/6efc3c2746eb4e139ed4d98fa4e34a89 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/6efc3c2746eb4e139ed4d98fa4e34a89 2023-03-26 13:55:05,545 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/f76eef3a4d974255af173cb87086709b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/f76eef3a4d974255af173cb87086709b 2023-03-26 13:55:05,546 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/829d66411492458fbee654c15917cb2a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/829d66411492458fbee654c15917cb2a 2023-03-26 13:55:05,546 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/5977fadd436d409694407398aae9fa0a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/5977fadd436d409694407398aae9fa0a 2023-03-26 13:55:05,547 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/aa5210527c4c4d21a57dcc011534dd94 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/aa5210527c4c4d21a57dcc011534dd94 2023-03-26 13:55:05,548 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/3919b03f597247989f7cd2251b22e10f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/3919b03f597247989f7cd2251b22e10f 2023-03-26 13:55:05,548 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/1a86daa1f73240928b2e9c7274c17364 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/1a86daa1f73240928b2e9c7274c17364 2023-03-26 13:55:05,549 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/b1f98ee81e864eb1abc477d99cfb27bc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/b1f98ee81e864eb1abc477d99cfb27bc 2023-03-26 13:55:05,550 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/144018325dc147149a5e6f75a7ca19ed to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/144018325dc147149a5e6f75a7ca19ed 2023-03-26 13:55:05,553 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6008e0a6bea7449f8625c72625dc4c36, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/e480469324b940c8904a17c4b3fcf834, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/4dd812c2ed71493b94b882ba86c7e80e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/581635bc5f984498b4206e10b075c888, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6a81af28407048e0a0c95610c62700ac, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/1acd1ef6fe024b8eb5959e32ff0dcbab, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6a7d051bac2c405c864836d2c8574185, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/0f2d23cf68dd45ed8ffac49e09cea5ae, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/b1700eb62f4a4598876c6eb2698fdc99, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/9e7979728998427f8ce1c1f0d20a46a1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/88f34535896d42deb59da0ef2d537894, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/b60a00807b874fb9a3dc1422af60a58e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/72c51e94e25f440e8e8537cdced41a44, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/abef16f9ca534e77b417bb184248502f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/70f822758e734ff587695467db4a643f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/e499f404792240a5859dd19e5976f4c8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/7887147a72774d52b81987cd81e53f41, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/0ff466cf95dd4420a65f0940af9844ee, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/b42e0c3dcd094913a06cc69db544c2c8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/c1575d18452f47c481210c1bbe7d0a89, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/a008a43063114e9486f17a2caa177a71, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/4428f115bd0f41c8934ac4ecb30c07b9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/af0d84c958c04c6097ad2eeec748c087, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/7c77dc0f5a344ceba06aef807b8db355, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/f0b83157744141e18c714b0ee1aa1590, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6f879f9ad97e49f9ae724909a83e8323, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/0f0253c6676945929a1fceb571eb9830, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/e483e3d32d4d45f2a08020fb8596bb35, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/5fb614a57799449db539e9f8c05af0bc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/a33199196eec484a888e16a5ea74f552, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/56bc42c7a691446ca3d8d19a6bf83a44, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/388dc639619740dea02f0cbc21dcb9b6, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/9fb797e8b9f64bf1a49743a0caa34924, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/8371a179f1c84987ba6e55fbfbe91ca5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/cc833ee2363b4e30b69b917e29ef73fa] to archive 2023-03-26 13:55:05,553 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 13:55:05,555 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6008e0a6bea7449f8625c72625dc4c36 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6008e0a6bea7449f8625c72625dc4c36 2023-03-26 13:55:05,556 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/e480469324b940c8904a17c4b3fcf834 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/e480469324b940c8904a17c4b3fcf834 2023-03-26 13:55:05,557 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/4dd812c2ed71493b94b882ba86c7e80e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/4dd812c2ed71493b94b882ba86c7e80e 2023-03-26 13:55:05,557 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/581635bc5f984498b4206e10b075c888 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/581635bc5f984498b4206e10b075c888 2023-03-26 13:55:05,558 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6a81af28407048e0a0c95610c62700ac to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6a81af28407048e0a0c95610c62700ac 2023-03-26 13:55:05,559 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/1acd1ef6fe024b8eb5959e32ff0dcbab to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/1acd1ef6fe024b8eb5959e32ff0dcbab 2023-03-26 13:55:05,560 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6a7d051bac2c405c864836d2c8574185 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6a7d051bac2c405c864836d2c8574185 2023-03-26 13:55:05,561 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/0f2d23cf68dd45ed8ffac49e09cea5ae to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/0f2d23cf68dd45ed8ffac49e09cea5ae 2023-03-26 13:55:05,562 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/b1700eb62f4a4598876c6eb2698fdc99 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/b1700eb62f4a4598876c6eb2698fdc99 2023-03-26 13:55:05,563 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/9e7979728998427f8ce1c1f0d20a46a1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/9e7979728998427f8ce1c1f0d20a46a1 2023-03-26 13:55:05,564 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/88f34535896d42deb59da0ef2d537894 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/88f34535896d42deb59da0ef2d537894 2023-03-26 13:55:05,564 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/b60a00807b874fb9a3dc1422af60a58e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/b60a00807b874fb9a3dc1422af60a58e 2023-03-26 13:55:05,565 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/72c51e94e25f440e8e8537cdced41a44 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/72c51e94e25f440e8e8537cdced41a44 2023-03-26 13:55:05,566 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/abef16f9ca534e77b417bb184248502f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/abef16f9ca534e77b417bb184248502f 2023-03-26 13:55:05,567 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/70f822758e734ff587695467db4a643f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/70f822758e734ff587695467db4a643f 2023-03-26 13:55:05,568 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/e499f404792240a5859dd19e5976f4c8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/e499f404792240a5859dd19e5976f4c8 2023-03-26 13:55:05,568 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/7887147a72774d52b81987cd81e53f41 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/7887147a72774d52b81987cd81e53f41 2023-03-26 13:55:05,569 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/0ff466cf95dd4420a65f0940af9844ee to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/0ff466cf95dd4420a65f0940af9844ee 2023-03-26 13:55:05,570 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/b42e0c3dcd094913a06cc69db544c2c8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/b42e0c3dcd094913a06cc69db544c2c8 2023-03-26 13:55:05,571 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/c1575d18452f47c481210c1bbe7d0a89 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/c1575d18452f47c481210c1bbe7d0a89 2023-03-26 13:55:05,571 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/a008a43063114e9486f17a2caa177a71 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/a008a43063114e9486f17a2caa177a71 2023-03-26 13:55:05,572 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/4428f115bd0f41c8934ac4ecb30c07b9 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/4428f115bd0f41c8934ac4ecb30c07b9 2023-03-26 13:55:05,573 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/af0d84c958c04c6097ad2eeec748c087 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/af0d84c958c04c6097ad2eeec748c087 2023-03-26 13:55:05,574 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/7c77dc0f5a344ceba06aef807b8db355 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/7c77dc0f5a344ceba06aef807b8db355 2023-03-26 13:55:05,575 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/f0b83157744141e18c714b0ee1aa1590 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/f0b83157744141e18c714b0ee1aa1590 2023-03-26 13:55:05,575 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6f879f9ad97e49f9ae724909a83e8323 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/6f879f9ad97e49f9ae724909a83e8323 2023-03-26 13:55:05,576 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/0f0253c6676945929a1fceb571eb9830 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/0f0253c6676945929a1fceb571eb9830 2023-03-26 13:55:05,577 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/e483e3d32d4d45f2a08020fb8596bb35 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/e483e3d32d4d45f2a08020fb8596bb35 2023-03-26 13:55:05,578 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/5fb614a57799449db539e9f8c05af0bc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/5fb614a57799449db539e9f8c05af0bc 2023-03-26 13:55:05,579 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/a33199196eec484a888e16a5ea74f552 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/a33199196eec484a888e16a5ea74f552 2023-03-26 13:55:05,580 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/56bc42c7a691446ca3d8d19a6bf83a44 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/56bc42c7a691446ca3d8d19a6bf83a44 2023-03-26 13:55:05,580 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/388dc639619740dea02f0cbc21dcb9b6 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/388dc639619740dea02f0cbc21dcb9b6 2023-03-26 13:55:05,581 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/9fb797e8b9f64bf1a49743a0caa34924 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/9fb797e8b9f64bf1a49743a0caa34924 2023-03-26 13:55:05,582 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/8371a179f1c84987ba6e55fbfbe91ca5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/8371a179f1c84987ba6e55fbfbe91ca5 2023-03-26 13:55:05,583 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/cc833ee2363b4e30b69b917e29ef73fa to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/cc833ee2363b4e30b69b917e29ef73fa 2023-03-26 13:55:05,586 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/0921b04106fd4d21a4d3497af4fea9cc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/96e17c02271f496d8758719958eae2af, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/3ed2e056725540f68c5dea343250b886, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/ab43724a9d4c4a6bb3a8174c81e1b760, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/eac81f885b594fe6a07d1be538bc716a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/eccb20ea57644adc9233ef9fa16359f1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/4c42adad613746bb855c77d7f8294287, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/15702b95af774f16ae59bb8e5fa4d72a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/4b6e1775206e41d29de269601dbb2d38, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/18a787d626d54e2fac78910fa620d2dc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/7790a4c70cca45e5ac2f5e3b452dbc66, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/42bb6fcc3f1e444883b8f85bddbead4a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/19c9379d43ec49f589ffae189e7ed5b5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/7af8ede8d1784c8a904e117af2dbcdcf, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/84429fa3df904ac1999a85eabbef1542, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/5458206b89154c32bdfdad871206592f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/032b3cfa9de6444a88c70615cc24f86c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/1011f920f18c4ee8bf48ad21ab339be5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/cd692c6c8d5d4927bd4ae05925a5b5ab, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/3b338a5c642d4bc0a12a07cd5fc81fb3, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/28af2500a09e4ea5866d7cc132772130, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/23d809eaabbb4d1d91dd3a7cc1f4adbd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/bdc4ec431bbe42cbb88f4bc668460266, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/1d499ff9941a4e179e08e415244ae56d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/ce1f8056976945f59f5bc04eab72a60a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/115957e9af88432882ee2cac516e39b4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/487af930246f45ba95d4ce23da53f8d4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/76e57f6e3f1f42b68df62bbbb4d026f8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/5772f12effba4cb493170910b9ebd7f2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/5bca65691396427a92a18ab83a4cd596, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/0805b8749f794b5da2b9f6d2ca516b34, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/204f163af093460198b70c28833d32f1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/c4f2fd4065c7467a9ca945974d060384, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/df93af02dc4641b6bb52a7bd9f2451b1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/f1610ef0301f4a1bae70d4aa581f82a1] to archive 2023-03-26 13:55:05,586 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 13:55:05,587 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/0921b04106fd4d21a4d3497af4fea9cc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/0921b04106fd4d21a4d3497af4fea9cc 2023-03-26 13:55:05,588 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/96e17c02271f496d8758719958eae2af to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/96e17c02271f496d8758719958eae2af 2023-03-26 13:55:05,589 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/3ed2e056725540f68c5dea343250b886 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/3ed2e056725540f68c5dea343250b886 2023-03-26 13:55:05,590 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/ab43724a9d4c4a6bb3a8174c81e1b760 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/ab43724a9d4c4a6bb3a8174c81e1b760 2023-03-26 13:55:05,591 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/eac81f885b594fe6a07d1be538bc716a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/eac81f885b594fe6a07d1be538bc716a 2023-03-26 13:55:05,591 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/eccb20ea57644adc9233ef9fa16359f1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/eccb20ea57644adc9233ef9fa16359f1 2023-03-26 13:55:05,592 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/4c42adad613746bb855c77d7f8294287 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/4c42adad613746bb855c77d7f8294287 2023-03-26 13:55:05,593 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/15702b95af774f16ae59bb8e5fa4d72a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/15702b95af774f16ae59bb8e5fa4d72a 2023-03-26 13:55:05,594 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/4b6e1775206e41d29de269601dbb2d38 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/4b6e1775206e41d29de269601dbb2d38 2023-03-26 13:55:05,595 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/18a787d626d54e2fac78910fa620d2dc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/18a787d626d54e2fac78910fa620d2dc 2023-03-26 13:55:05,596 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/7790a4c70cca45e5ac2f5e3b452dbc66 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/7790a4c70cca45e5ac2f5e3b452dbc66 2023-03-26 13:55:05,596 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/42bb6fcc3f1e444883b8f85bddbead4a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/42bb6fcc3f1e444883b8f85bddbead4a 2023-03-26 13:55:05,597 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/19c9379d43ec49f589ffae189e7ed5b5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/19c9379d43ec49f589ffae189e7ed5b5 2023-03-26 13:55:05,598 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/7af8ede8d1784c8a904e117af2dbcdcf to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/7af8ede8d1784c8a904e117af2dbcdcf 2023-03-26 13:55:05,599 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/84429fa3df904ac1999a85eabbef1542 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/84429fa3df904ac1999a85eabbef1542 2023-03-26 13:55:05,600 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/5458206b89154c32bdfdad871206592f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/5458206b89154c32bdfdad871206592f 2023-03-26 13:55:05,601 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/032b3cfa9de6444a88c70615cc24f86c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/032b3cfa9de6444a88c70615cc24f86c 2023-03-26 13:55:05,601 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/1011f920f18c4ee8bf48ad21ab339be5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/1011f920f18c4ee8bf48ad21ab339be5 2023-03-26 13:55:05,602 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/cd692c6c8d5d4927bd4ae05925a5b5ab to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/cd692c6c8d5d4927bd4ae05925a5b5ab 2023-03-26 13:55:05,603 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/3b338a5c642d4bc0a12a07cd5fc81fb3 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/3b338a5c642d4bc0a12a07cd5fc81fb3 2023-03-26 13:55:05,604 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/28af2500a09e4ea5866d7cc132772130 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/28af2500a09e4ea5866d7cc132772130 2023-03-26 13:55:05,605 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/23d809eaabbb4d1d91dd3a7cc1f4adbd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/23d809eaabbb4d1d91dd3a7cc1f4adbd 2023-03-26 13:55:05,606 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/bdc4ec431bbe42cbb88f4bc668460266 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/bdc4ec431bbe42cbb88f4bc668460266 2023-03-26 13:55:05,607 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/1d499ff9941a4e179e08e415244ae56d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/1d499ff9941a4e179e08e415244ae56d 2023-03-26 13:55:05,607 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/ce1f8056976945f59f5bc04eab72a60a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/ce1f8056976945f59f5bc04eab72a60a 2023-03-26 13:55:05,608 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/115957e9af88432882ee2cac516e39b4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/115957e9af88432882ee2cac516e39b4 2023-03-26 13:55:05,609 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/487af930246f45ba95d4ce23da53f8d4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/487af930246f45ba95d4ce23da53f8d4 2023-03-26 13:55:05,610 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/76e57f6e3f1f42b68df62bbbb4d026f8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/76e57f6e3f1f42b68df62bbbb4d026f8 2023-03-26 13:55:05,610 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/5772f12effba4cb493170910b9ebd7f2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/5772f12effba4cb493170910b9ebd7f2 2023-03-26 13:55:05,611 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/5bca65691396427a92a18ab83a4cd596 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/5bca65691396427a92a18ab83a4cd596 2023-03-26 13:55:05,612 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/0805b8749f794b5da2b9f6d2ca516b34 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/0805b8749f794b5da2b9f6d2ca516b34 2023-03-26 13:55:05,613 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/204f163af093460198b70c28833d32f1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/204f163af093460198b70c28833d32f1 2023-03-26 13:55:05,614 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/c4f2fd4065c7467a9ca945974d060384 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/c4f2fd4065c7467a9ca945974d060384 2023-03-26 13:55:05,614 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/df93af02dc4641b6bb52a7bd9f2451b1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/df93af02dc4641b6bb52a7bd9f2451b1 2023-03-26 13:55:05,615 DEBUG [StoreCloser-TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/f1610ef0301f4a1bae70d4aa581f82a1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/f1610ef0301f4a1bae70d4aa581f82a1 2023-03-26 13:55:05,618 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/recovered.edits/606.seqid, newMaxSeqId=606, maxSeqId=1 2023-03-26 13:55:05,620 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d. 2023-03-26 13:55:05,620 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a59be4541fa5742a2bd937206749175d: 2023-03-26 13:55:05,621 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a59be4541fa5742a2bd937206749175d 2023-03-26 13:55:05,621 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=a59be4541fa5742a2bd937206749175d, regionState=CLOSED 2023-03-26 13:55:05,622 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838905621"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679838905621"}]},"ts":"1679838905621"} 2023-03-26 13:55:05,623 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-03-26 13:55:05,624 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure a59be4541fa5742a2bd937206749175d, server=jenkins-hbase4.apache.org,42517,1679838814721 in 1.5080 sec 2023-03-26 13:55:05,625 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-03-26 13:55:05,625 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a59be4541fa5742a2bd937206749175d, UNASSIGN in 1.5130 sec 2023-03-26 13:55:05,625 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838905625"}]},"ts":"1679838905625"} 2023-03-26 13:55:05,626 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-26 13:55:05,627 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-26 13:55:05,629 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5220 sec 2023-03-26 13:55:06,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-26 13:55:06,212 INFO [Listener at localhost/41531] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-03-26 13:55:06,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-26 13:55:06,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:55:06,215 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:55:06,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-26 13:55:06,216 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:55:06,219 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d 2023-03-26 13:55:06,221 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A, FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B, FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C, FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/recovered.edits] 2023-03-26 13:55:06,223 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/e9d50c6593b84236bd6cb16e23574f75 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/e9d50c6593b84236bd6cb16e23574f75 2023-03-26 13:55:06,224 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/f1c4b3e7d9554b78bf06df9084c30c55 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/A/f1c4b3e7d9554b78bf06df9084c30c55 2023-03-26 13:55:06,226 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/10a34a1e0342450e997f23a1d72cbf87 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/10a34a1e0342450e997f23a1d72cbf87 2023-03-26 13:55:06,227 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/2a2fd500acf84bc5b5fe1b5fda81df8f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/B/2a2fd500acf84bc5b5fe1b5fda81df8f 2023-03-26 13:55:06,229 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/a0684935a10c4689b8cfec54179f4821 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/a0684935a10c4689b8cfec54179f4821 2023-03-26 13:55:06,230 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/c01b76e6330446f98eb5886f2c06dbcd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/C/c01b76e6330446f98eb5886f2c06dbcd 2023-03-26 13:55:06,232 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/recovered.edits/606.seqid to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d/recovered.edits/606.seqid 2023-03-26 13:55:06,232 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a59be4541fa5742a2bd937206749175d 2023-03-26 13:55:06,233 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 13:55:06,234 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:55:06,235 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-26 13:55:06,237 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-26 13:55:06,238 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:55:06,238 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-26 13:55:06,238 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679838906238"}]},"ts":"9223372036854775807"} 2023-03-26 13:55:06,241 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-26 13:55:06,241 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a59be4541fa5742a2bd937206749175d, NAME => 'TestAcidGuarantees,,1679838876143.a59be4541fa5742a2bd937206749175d.', STARTKEY => '', ENDKEY => ''}] 2023-03-26 13:55:06,241 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-26 13:55:06,241 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679838906241"}]},"ts":"9223372036854775807"} 2023-03-26 13:55:06,242 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-26 13:55:06,244 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:55:06,245 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 31 msec 2023-03-26 13:55:06,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-26 13:55:06,316 INFO [Listener at localhost/41531] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-03-26 13:55:06,327 INFO [Listener at localhost/41531] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 232), OpenFileDescriptor=529 (was 591), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=401 (was 472), ProcessCount=172 (was 172), AvailableMemoryMB=8749 (was 8821) 2023-03-26 13:55:06,338 INFO [Listener at localhost/41531] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=529, MaxFileDescriptor=60000, SystemLoadAverage=401, ProcessCount=172, AvailableMemoryMB=8748 2023-03-26 13:55:06,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-26 13:55:06,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 13:55:06,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-26 13:55:06,342 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-26 13:55:06,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-03-26 13:55:06,342 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-26 13:55:06,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-26 13:55:06,343 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:06,344 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0 empty. 2023-03-26 13:55:06,344 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:06,344 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 13:55:06,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-26 13:55:06,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-26 13:55:06,751 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-26 13:55:06,752 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a9871eaedebb91319d17cf11612be1e0, NAME => 'TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp 2023-03-26 13:55:06,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-26 13:55:07,157 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:55:07,157 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a9871eaedebb91319d17cf11612be1e0, disabling compactions & flushes 2023-03-26 13:55:07,157 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:07,157 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:07,157 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. after waiting 0 ms 2023-03-26 13:55:07,157 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:07,158 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:07,158 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:07,160 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-26 13:55:07,161 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838907160"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679838907160"}]},"ts":"1679838907160"} 2023-03-26 13:55:07,161 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-26 13:55:07,162 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-26 13:55:07,162 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838907162"}]},"ts":"1679838907162"} 2023-03-26 13:55:07,163 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-26 13:55:07,166 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9871eaedebb91319d17cf11612be1e0, ASSIGN}] 2023-03-26 13:55:07,167 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9871eaedebb91319d17cf11612be1e0, ASSIGN 2023-03-26 13:55:07,168 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9871eaedebb91319d17cf11612be1e0, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42517,1679838814721; forceNewPlan=false, retain=false 2023-03-26 13:55:07,319 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=a9871eaedebb91319d17cf11612be1e0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:07,319 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838907319"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838907319"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838907319"}]},"ts":"1679838907319"} 2023-03-26 13:55:07,320 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:55:07,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-26 13:55:07,475 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:07,475 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a9871eaedebb91319d17cf11612be1e0, NAME => 'TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0.', STARTKEY => '', ENDKEY => ''} 2023-03-26 13:55:07,475 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:07,475 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:55:07,475 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:07,475 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:07,476 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:07,478 DEBUG [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A 2023-03-26 13:55:07,478 DEBUG [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A 2023-03-26 13:55:07,478 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:55:07,478 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a9871eaedebb91319d17cf11612be1e0 columnFamilyName A 2023-03-26 13:55:07,479 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] regionserver.HStore(310): Store=a9871eaedebb91319d17cf11612be1e0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:55:07,479 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:07,480 DEBUG [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B 2023-03-26 13:55:07,480 DEBUG [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B 2023-03-26 13:55:07,480 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:55:07,480 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a9871eaedebb91319d17cf11612be1e0 columnFamilyName B 2023-03-26 13:55:07,481 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] regionserver.HStore(310): Store=a9871eaedebb91319d17cf11612be1e0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:55:07,481 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:07,482 DEBUG [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C 2023-03-26 13:55:07,482 DEBUG [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C 2023-03-26 13:55:07,482 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:55:07,482 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a9871eaedebb91319d17cf11612be1e0 columnFamilyName C 2023-03-26 13:55:07,483 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] regionserver.HStore(310): Store=a9871eaedebb91319d17cf11612be1e0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:55:07,483 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:07,484 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:07,484 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:07,485 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 13:55:07,486 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:07,488 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-26 13:55:07,488 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a9871eaedebb91319d17cf11612be1e0; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66864393, jitterRate=-0.0036429017782211304}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 13:55:07,488 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:07,489 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., pid=37, masterSystemTime=1679838907471 2023-03-26 13:55:07,490 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:07,490 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:07,490 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=a9871eaedebb91319d17cf11612be1e0, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:07,490 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838907490"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679838907490"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679838907490"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679838907490"}]},"ts":"1679838907490"} 2023-03-26 13:55:07,492 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-03-26 13:55:07,492 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 in 171 msec 2023-03-26 13:55:07,493 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-03-26 13:55:07,493 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9871eaedebb91319d17cf11612be1e0, ASSIGN in 326 msec 2023-03-26 13:55:07,494 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-26 13:55:07,494 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838907494"}]},"ts":"1679838907494"} 2023-03-26 13:55:07,494 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-26 13:55:07,497 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-26 13:55:07,498 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1570 sec 2023-03-26 13:55:08,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-26 13:55:08,447 INFO [Listener at localhost/41531] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-03-26 13:55:08,448 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x494529f9 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:08,452 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@118e131f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:08,453 DEBUG [Listener at localhost/41531] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:08,455 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53788, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:08,456 DEBUG [Listener at localhost/41531] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-26 13:55:08,457 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:42508, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-26 13:55:08,458 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-26 13:55:08,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 13:55:08,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-26 13:55:08,868 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-26 13:55:08,869 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-26 13:55:08,869 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-26 13:55:08,870 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-26 13:55:08,872 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9871eaedebb91319d17cf11612be1e0, REOPEN/MOVE}] 2023-03-26 13:55:08,873 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9871eaedebb91319d17cf11612be1e0, REOPEN/MOVE 2023-03-26 13:55:08,874 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a9871eaedebb91319d17cf11612be1e0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:08,874 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838908874"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838908874"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838908874"}]},"ts":"1679838908874"} 2023-03-26 13:55:08,875 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:55:09,026 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:09,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a9871eaedebb91319d17cf11612be1e0, disabling compactions & flushes 2023-03-26 13:55:09,027 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:09,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:09,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. after waiting 0 ms 2023-03-26 13:55:09,027 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:09,030 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-26 13:55:09,031 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:09,031 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:09,031 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: a9871eaedebb91319d17cf11612be1e0 to self. 2023-03-26 13:55:09,032 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:09,032 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a9871eaedebb91319d17cf11612be1e0, regionState=CLOSED 2023-03-26 13:55:09,032 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838909032"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679838909032"}]},"ts":"1679838909032"} 2023-03-26 13:55:09,034 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-03-26 13:55:09,034 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 in 158 msec 2023-03-26 13:55:09,034 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9871eaedebb91319d17cf11612be1e0, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,42517,1679838814721; forceNewPlan=false, retain=true 2023-03-26 13:55:09,185 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a9871eaedebb91319d17cf11612be1e0, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,185 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838909185"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838909185"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838909185"}]},"ts":"1679838909185"} 2023-03-26 13:55:09,186 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:55:09,340 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:09,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a9871eaedebb91319d17cf11612be1e0, NAME => 'TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0.', STARTKEY => '', ENDKEY => ''} 2023-03-26 13:55:09,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:09,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:55:09,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:09,341 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:09,342 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:09,343 DEBUG [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A 2023-03-26 13:55:09,343 DEBUG [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A 2023-03-26 13:55:09,344 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:55:09,344 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a9871eaedebb91319d17cf11612be1e0 columnFamilyName A 2023-03-26 13:55:09,345 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] regionserver.HStore(310): Store=a9871eaedebb91319d17cf11612be1e0/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:55:09,345 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:09,346 DEBUG [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B 2023-03-26 13:55:09,346 DEBUG [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B 2023-03-26 13:55:09,346 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:55:09,346 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a9871eaedebb91319d17cf11612be1e0 columnFamilyName B 2023-03-26 13:55:09,347 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] regionserver.HStore(310): Store=a9871eaedebb91319d17cf11612be1e0/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:55:09,347 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:09,347 DEBUG [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C 2023-03-26 13:55:09,347 DEBUG [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C 2023-03-26 13:55:09,347 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:55:09,348 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a9871eaedebb91319d17cf11612be1e0 columnFamilyName C 2023-03-26 13:55:09,348 INFO [StoreOpener-a9871eaedebb91319d17cf11612be1e0-1] regionserver.HStore(310): Store=a9871eaedebb91319d17cf11612be1e0/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:55:09,348 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:09,349 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:09,350 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:09,351 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 13:55:09,352 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:09,353 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened a9871eaedebb91319d17cf11612be1e0; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59276666, jitterRate=-0.11670884490013123}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 13:55:09,353 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:09,353 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., pid=42, masterSystemTime=1679838909338 2023-03-26 13:55:09,355 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:09,355 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:09,355 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=a9871eaedebb91319d17cf11612be1e0, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,355 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838909355"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679838909355"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679838909355"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679838909355"}]},"ts":"1679838909355"} 2023-03-26 13:55:09,357 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-03-26 13:55:09,357 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 in 170 msec 2023-03-26 13:55:09,358 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-03-26 13:55:09,358 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9871eaedebb91319d17cf11612be1e0, REOPEN/MOVE in 485 msec 2023-03-26 13:55:09,360 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-03-26 13:55:09,360 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 489 msec 2023-03-26 13:55:09,361 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 901 msec 2023-03-26 13:55:09,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-03-26 13:55:09,363 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x4d800352 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:09,368 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2deaf1fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:09,368 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x2329054b to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:09,372 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e7e18ef, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:09,373 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x30951255 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:09,383 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5b7d4a51, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:09,383 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x5baa6768 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:09,387 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4335c1f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:09,387 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x63a535f3 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:09,391 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6bbc9fc8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:09,391 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x678b4c3d to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:09,394 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6846dddb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:09,394 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x43736f95 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:09,398 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ea43962, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:09,398 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x72eaaf6e to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:09,406 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2abd7961, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:09,406 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x4130667f to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:09,410 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@660630e8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:09,411 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x3b73a361 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:09,414 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@793239ee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:09,419 DEBUG [hconnection-0x4263ecda-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:09,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:09,420 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53790, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:09,425 DEBUG [hconnection-0xf84db62-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:09,426 DEBUG [hconnection-0x38c23d6c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:09,426 DEBUG [hconnection-0x24f9e6a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:09,426 DEBUG [hconnection-0x3725d520-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:09,428 DEBUG [hconnection-0xfea9537-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:09,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:09,428 DEBUG [hconnection-0x364c0e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:09,428 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53794, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:09,428 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53804, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:09,428 DEBUG [hconnection-0x1f5e9984-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:09,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:09,429 DEBUG [hconnection-0x7e93f180-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:09,428 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53808, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:09,429 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:09,429 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:09,429 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:09,429 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:09,433 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53818, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:09,433 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53826, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:09,434 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53840, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:09,434 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53834, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:09,434 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:09,435 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:09,438 DEBUG [hconnection-0x3d4527c3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:09,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:09,438 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53846, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:09,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:09,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:09,440 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,440 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:09,440 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53856, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:09,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:09,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:09,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:09,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:09,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:09,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:09,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:09,445 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:09,445 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:09,445 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:09,446 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:09,449 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:09,449 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:09,449 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:09,449 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-26 13:55:09,450 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:09,450 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:09,450 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:09,450 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:09,451 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:09,451 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:09,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:09,452 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:09,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679838909457, max timestamp=1679838909464Num uniques 859; 2023-03-26 13:55:09,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679838909457, max timestamp=1679838909464Num uniques 859; 2023-03-26 13:55:09,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 13:55:09,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 13:55:09,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1679838909457, max timestamp=1679838909464Num uniques 859; 2023-03-26 13:55:09,467 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 13:55:09,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:09,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:09,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838969477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1679838969479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:09,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:09,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53804 deadline: 1679838969479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838969479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:09,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838969479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:09,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:09,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838969581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838969581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:09,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:09,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53804 deadline: 1679838969581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838969581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:09,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1679838969581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:09,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838969786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:09,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838969786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:09,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838969786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:09,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53804 deadline: 1679838969787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:09,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1679838969787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:09,881 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326421467581688423ca459a526632cea5c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326421467581688423ca459a526632cea5c 2023-03-26 13:55:09,887 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=13, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/5262cda5f8fa47bc874a077bb9d8d2b6 2023-03-26 13:55:10,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:10,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838970091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:10,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53804 deadline: 1679838970091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:10,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1679838970091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:10,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838970092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:10,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838970092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,296 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/371e209d940147d692de96e5d483aa9f 2023-03-26 13:55:10,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:10,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838970596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:10,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53804 deadline: 1679838970600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:10,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1679838970600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,604 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:10,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838970601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:10,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838970602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,706 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/672ce41860874484b85de7444618af7c 2023-03-26 13:55:10,710 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/5262cda5f8fa47bc874a077bb9d8d2b6 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5262cda5f8fa47bc874a077bb9d8d2b6 2023-03-26 13:55:10,713 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5262cda5f8fa47bc874a077bb9d8d2b6, entries=100, sequenceid=13, filesize=18.4 K 2023-03-26 13:55:10,714 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/371e209d940147d692de96e5d483aa9f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/371e209d940147d692de96e5d483aa9f 2023-03-26 13:55:10,718 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/371e209d940147d692de96e5d483aa9f, entries=100, sequenceid=13, filesize=9.3 K 2023-03-26 13:55:10,718 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/672ce41860874484b85de7444618af7c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/672ce41860874484b85de7444618af7c 2023-03-26 13:55:10,721 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/672ce41860874484b85de7444618af7c, entries=100, sequenceid=13, filesize=9.3 K 2023-03-26 13:55:10,722 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=154.31 KB/158010 for a9871eaedebb91319d17cf11612be1e0 in 1273ms, sequenceid=13, compaction requested=false 2023-03-26 13:55:10,722 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-26 13:55:10,723 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:10,723 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:10,723 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:10,723 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:10,723 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,723 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:10,723 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:10,725 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:10,725 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:10,725 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:10,725 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:10,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:10,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:10,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:10,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:10,726 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:10,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6750ac04[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:10,726 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:10,727 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:10,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:10,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:10,727 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:10,727 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:10,727 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:10,727 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,728 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:10,729 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:10,729 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,729 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:10,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:10,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:10,729 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:10,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:10,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:10,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:10,731 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:10,731 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:10,732 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:10,732 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:10,732 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:10,738 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:10,738 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:10,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:10,738 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:10,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:10,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:10,738 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:10,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:10,739 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:10,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:10,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,741 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,741 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:10,741 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:10,741 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,741 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:10,741 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:10,742 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:10,742 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:10,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:10,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:10,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:10,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:10,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:10,742 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:10,742 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:10,742 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:10,742 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:10,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:10,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:10,842 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:10,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:10,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:10,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:10,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:10,846 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:10,846 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:10,846 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:10,846 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:10,846 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:10,846 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:10,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:10,847 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:10,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:10,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:10,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:10,848 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:10,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:10,849 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:10,849 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:10,849 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:10,849 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:10,849 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:10,850 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:10,850 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:10,850 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:10,850 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:10,850 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=154.31 KB heapSize=430.69 KB 2023-03-26 13:55:10,851 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:10,851 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:10,851 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:10,851 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:10,851 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:10,851 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:11,142 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-26 13:55:11,263 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303261f5fd12b940d4e009eca7d25c470eae0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303261f5fd12b940d4e009eca7d25c470eae0 2023-03-26 13:55:11,268 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/fd97c05aaf5446da983dfb9bf969028f 2023-03-26 13:55:11,277 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/5bdb5ef59a8b4309b388d5a66ff8f97a 2023-03-26 13:55:11,290 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/2822f467167142cf9e48d96f7b54001b 2023-03-26 13:55:11,293 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/fd97c05aaf5446da983dfb9bf969028f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/fd97c05aaf5446da983dfb9bf969028f 2023-03-26 13:55:11,296 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/fd97c05aaf5446da983dfb9bf969028f, entries=150, sequenceid=50, filesize=25.1 K 2023-03-26 13:55:11,297 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/5bdb5ef59a8b4309b388d5a66ff8f97a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/5bdb5ef59a8b4309b388d5a66ff8f97a 2023-03-26 13:55:11,304 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/5bdb5ef59a8b4309b388d5a66ff8f97a, entries=150, sequenceid=50, filesize=11.6 K 2023-03-26 13:55:11,304 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/2822f467167142cf9e48d96f7b54001b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/2822f467167142cf9e48d96f7b54001b 2023-03-26 13:55:11,308 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/2822f467167142cf9e48d96f7b54001b, entries=150, sequenceid=50, filesize=11.6 K 2023-03-26 13:55:11,309 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.64 KB/440976, currentSize=0 B/0 for a9871eaedebb91319d17cf11612be1e0 in 459ms, sequenceid=50, compaction requested=false 2023-03-26 13:55:11,309 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:11,309 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:11,309 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:11,309 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:11,309 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,309 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:11,309 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:11,312 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,316 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:11,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:11,316 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,316 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:11,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:11,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:11,318 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:11,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13fe9134[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:11,318 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,319 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,319 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:11,319 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:11,319 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,319 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:11,319 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:11,320 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,320 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:11,320 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:11,320 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:11,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:11,323 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:11,323 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:11,323 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:11,323 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:11,323 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:11,324 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,324 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,325 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:11,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:11,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,325 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:11,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:11,326 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,329 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:11,330 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:11,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:11,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:11,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:11,330 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:11,330 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:11,330 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:11,330 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:11,330 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:11,330 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:11,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:11,330 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,330 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,330 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,330 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,430 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:11,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:11,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:11,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:11,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:11,436 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:11,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:11,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:11,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:11,437 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,437 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:11,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:11,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:11,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:11,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:11,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:11,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:11,439 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:11,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:11,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:11,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:11,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:11,440 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:11,440 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:11,440 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:11,440 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:11,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:11,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:11,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:11,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:11,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,441 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:11,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:11,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:11,442 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,442 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:11,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:11,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:11,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:11,445 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:11,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17a531aa[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:11,445 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,446 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,446 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:11,446 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:11,446 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:11,446 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,446 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:11,448 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:11,448 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:11,448 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,448 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:11,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:11,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:11,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:11,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:11,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:11,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:11,451 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:11,451 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:11,451 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:11,451 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:11,451 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:11,453 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,453 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:11,453 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,453 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,455 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,457 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:11,457 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:11,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:11,457 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,457 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,457 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,457 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,457 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:11,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:11,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:11,457 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,458 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,457 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:11,458 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:11,458 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:11,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:11,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:11,558 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:11,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:11,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:11,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:11,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:11,561 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:11,562 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:11,562 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:11,562 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:11,562 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,562 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,563 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:11,563 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:11,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:11,564 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,564 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:11,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:11,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:11,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:11,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:11,565 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:11,565 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:11,565 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:11,565 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:11,565 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:11,565 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:11,565 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:11,566 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:11,566 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:11,566 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:11,566 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:11,566 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,566 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:11,566 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:11,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,569 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:11,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:11,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:11,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:11,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:11,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:11,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:11,571 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:11,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48dd4f89[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:11,571 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,572 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,572 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:11,572 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:11,572 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,572 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:11,572 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:11,573 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:11,573 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:11,573 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:11,573 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:11,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:11,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:11,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:11,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:11,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:11,576 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:11,576 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:11,576 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:11,576 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:11,576 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:11,577 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,577 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:11,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,578 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:11,578 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:11,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,579 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,585 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,585 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:11,585 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,586 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,586 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,586 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,586 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,586 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:11,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:11,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:11,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:11,586 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:11,586 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:11,586 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:11,586 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:11,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:11,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:11,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:11,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:11,675 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:55:11,676 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:11,676 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:11,676 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:11,676 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:11,676 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:11,676 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:11,686 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:11,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:11,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:11,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:11,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:11,690 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:11,690 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:11,690 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:11,690 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:11,691 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,691 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:11,692 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:11,692 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,692 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:11,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:11,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:11,693 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:11,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:11,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:11,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:11,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:11,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:11,694 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:11,694 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:11,694 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:11,694 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:11,694 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:11,694 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:11,694 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:11,694 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,694 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:11,694 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:11,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,695 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,696 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:11,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:11,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:11,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:11,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:11,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:11,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:11,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a0fea1a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:11,698 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:11,698 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,701 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:11,701 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:11,701 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:11,701 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,701 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:11,702 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,702 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:11,703 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:11,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:11,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:11,703 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:11,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:11,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:11,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:11,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:11,705 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:11,706 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:11,706 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:11,706 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:11,706 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:11,707 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:11,707 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,707 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:11,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:11,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:11,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:11,708 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:11,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,710 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:11,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:11,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838971706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838911701, max timestamp=1679838911710Num uniques 859; 2023-03-26 13:55:11,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:55:11,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838911701, max timestamp=1679838911710Num uniques 859; 2023-03-26 13:55:11,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:55:11,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838911701, max timestamp=1679838911710Num uniques 859; 2023-03-26 13:55:11,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:55:11,714 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,714 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,714 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,714 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,714 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,714 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:11,714 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:11,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:11,714 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:11,715 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:11,714 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:11,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:11,715 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:11,715 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:11,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:11,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:11,741 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:11,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838971732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:11,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53804 deadline: 1679838971733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:11,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838971734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:11,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1679838971744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,815 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:11,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:11,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:11,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838971813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:11,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:11,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:11,819 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:11,819 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:11,819 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:11,819 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:11,819 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,820 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,822 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:11,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:11,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:11,822 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,822 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:11,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:11,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,823 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:11,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:11,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:11,823 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:11,823 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:11,823 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:11,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:11,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:11,824 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:11,824 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:11,824 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:11,824 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:11,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:11,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:11,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:11,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:11,825 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:11,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:11,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:11,826 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,826 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:11,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:11,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:11,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:11,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d9fa95e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:11,828 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:11,828 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,829 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,829 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:11,829 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:11,829 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:11,829 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,829 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:11,831 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,831 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:11,831 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:11,831 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:11,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:11,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:11,833 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:11,833 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:11,833 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:11,833 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:11,833 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:11,835 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,835 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:11,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,835 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:11,836 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,838 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,840 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:11,840 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:11,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:11,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:11,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:11,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:11,841 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,841 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:11,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:11,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:11,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:11,841 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,842 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:11,842 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:11,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:11,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838971842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:11,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53804 deadline: 1679838971842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,847 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:11,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838971844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:11,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1679838971849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,942 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:11,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:11,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:11,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:11,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:11,945 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:11,945 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:11,945 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:11,945 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:11,946 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,946 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,947 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:11,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:11,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:11,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:11,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:11,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:11,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:11,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:11,948 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:11,948 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:11,948 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:11,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:11,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:11,949 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:11,949 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:11,949 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:11,949 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:11,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:11,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:11,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:11,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:11,950 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,950 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:11,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:11,950 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,950 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:11,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:11,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:11,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:11,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:11,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7309f926[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:11,952 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:11,952 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,954 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,954 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:11,955 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:11,955 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,955 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:11,955 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:11,956 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,956 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:11,956 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:11,956 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:11,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:11,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:11,959 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:11,959 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:11,959 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:11,959 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:11,959 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:11,960 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,960 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,961 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:11,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:11,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:11,961 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:11,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:11,962 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:11,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,968 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,968 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,968 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:11,968 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:11,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:11,968 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:11,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:11,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:11,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:11,968 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:11,969 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:11,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:11,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:11,969 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,969 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:11,969 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:11,969 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:11,969 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:12,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838972018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838972047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53804 deadline: 1679838972047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838972049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1679838972057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,069 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:12,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:12,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:12,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:12,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:12,072 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:12,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:12,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:12,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:12,073 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:12,073 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:12,074 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:12,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:12,074 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:12,074 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:12,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:12,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:12,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:12,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:12,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:12,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:12,075 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:12,075 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:12,075 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:12,075 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:12,075 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:12,075 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:12,075 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:12,076 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:12,076 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:12,076 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:12,076 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:12,076 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,076 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:12,076 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:12,077 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,077 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:12,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:12,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:12,077 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:12,077 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:12,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:12,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:12,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:12,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:12,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:12,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@518a51c9[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:12,079 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:12,079 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:12,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:12,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:12,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:12,085 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:12,085 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,085 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:12,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:12,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:12,086 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:12,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:12,088 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:12,089 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:12,089 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:12,089 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:12,089 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:12,090 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:12,090 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:12,090 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:12,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:12,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:12,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:12,091 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:12,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:12,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,094 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,094 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326da5e760513e84648abf523e6552632ba to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326da5e760513e84648abf523e6552632ba 2023-03-26 13:55:12,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:12,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,098 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,098 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:12,098 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:12,098 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:12,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:12,098 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:12,098 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:12,098 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:12,098 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:12,099 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:12,099 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:12,099 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/4a5a4ebfcb1b43a991d0ffa498dd46c3 2023-03-26 13:55:12,100 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:12,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:12,100 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:12,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:12,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:12,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:12,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:12,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:12,125 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/4cdcf89ed49a4db4880f28ea8e5d02a3 2023-03-26 13:55:12,135 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/62f5cd7dc2354cde9befe212ed428027 2023-03-26 13:55:12,138 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/4a5a4ebfcb1b43a991d0ffa498dd46c3 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/4a5a4ebfcb1b43a991d0ffa498dd46c3 2023-03-26 13:55:12,141 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/4a5a4ebfcb1b43a991d0ffa498dd46c3, entries=400, sequenceid=63, filesize=59.0 K 2023-03-26 13:55:12,142 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/4cdcf89ed49a4db4880f28ea8e5d02a3 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/4cdcf89ed49a4db4880f28ea8e5d02a3 2023-03-26 13:55:12,145 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/4cdcf89ed49a4db4880f28ea8e5d02a3, entries=150, sequenceid=63, filesize=11.6 K 2023-03-26 13:55:12,146 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/62f5cd7dc2354cde9befe212ed428027 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/62f5cd7dc2354cde9befe212ed428027 2023-03-26 13:55:12,150 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/62f5cd7dc2354cde9befe212ed428027, entries=150, sequenceid=63, filesize=11.6 K 2023-03-26 13:55:12,151 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a9871eaedebb91319d17cf11612be1e0 in 476ms, sequenceid=63, compaction requested=true 2023-03-26 13:55:12,151 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:12,151 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:12,151 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:12,151 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:12,151 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:12,151 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:12,152 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 104998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:12,152 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a9871eaedebb91319d17cf11612be1e0/A is initiating minor compaction (all files) 2023-03-26 13:55:12,152 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:12,152 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a9871eaedebb91319d17cf11612be1e0/A in TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:12,152 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a9871eaedebb91319d17cf11612be1e0/B is initiating minor compaction (all files) 2023-03-26 13:55:12,152 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5262cda5f8fa47bc874a077bb9d8d2b6, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/fd97c05aaf5446da983dfb9bf969028f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/4a5a4ebfcb1b43a991d0ffa498dd46c3] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp, totalSize=102.5 K 2023-03-26 13:55:12,152 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9871eaedebb91319d17cf11612be1e0/B in TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:12,152 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/371e209d940147d692de96e5d483aa9f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/5bdb5ef59a8b4309b388d5a66ff8f97a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/4cdcf89ed49a4db4880f28ea8e5d02a3] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp, totalSize=32.5 K 2023-03-26 13:55:12,152 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 5262cda5f8fa47bc874a077bb9d8d2b6, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679838909445 2023-03-26 13:55:12,152 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 371e209d940147d692de96e5d483aa9f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679838909445 2023-03-26 13:55:12,153 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting fd97c05aaf5446da983dfb9bf969028f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679838909477 2023-03-26 13:55:12,153 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 5bdb5ef59a8b4309b388d5a66ff8f97a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679838909477 2023-03-26 13:55:12,153 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 4a5a4ebfcb1b43a991d0ffa498dd46c3, keycount=400, bloomtype=ROW, size=59.0 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679838911653 2023-03-26 13:55:12,153 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 4cdcf89ed49a4db4880f28ea8e5d02a3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679838911653 2023-03-26 13:55:12,181 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a9871eaedebb91319d17cf11612be1e0#B#compaction#360 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:12,191 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a9871eaedebb91319d17cf11612be1e0#A#compaction#361 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:12,199 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:12,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:12,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:12,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:12,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:12,202 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:12,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:12,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:12,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:12,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:12,203 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:12,209 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:12,209 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:12,210 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,210 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:12,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:12,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:12,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:12,211 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:12,211 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:12,211 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:12,211 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:12,211 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:12,212 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:12,212 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:12,212 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:12,212 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:12,212 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-26 13:55:12,213 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:12,213 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:12,213 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:12,213 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:12,213 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:12,213 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:12,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:12,332 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:12,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53804 deadline: 1679838972358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838972362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838972363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,373 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838972364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1679838972367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,475 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838972472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838972474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838972474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,597 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/dcfb160bfa4e47d58fb8acdbfc33eb86 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/dcfb160bfa4e47d58fb8acdbfc33eb86 2023-03-26 13:55:12,605 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9871eaedebb91319d17cf11612be1e0/B of a9871eaedebb91319d17cf11612be1e0 into dcfb160bfa4e47d58fb8acdbfc33eb86(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:12,605 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:12,605 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., storeName=a9871eaedebb91319d17cf11612be1e0/B, priority=13, startTime=1679838912151; duration=0sec 2023-03-26 13:55:12,605 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:12,605 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:12,608 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:12,609 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a9871eaedebb91319d17cf11612be1e0/C is initiating minor compaction (all files) 2023-03-26 13:55:12,609 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9871eaedebb91319d17cf11612be1e0/C in TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:12,609 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/672ce41860874484b85de7444618af7c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/2822f467167142cf9e48d96f7b54001b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/62f5cd7dc2354cde9befe212ed428027] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp, totalSize=32.5 K 2023-03-26 13:55:12,609 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 672ce41860874484b85de7444618af7c, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679838909445 2023-03-26 13:55:12,609 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 2822f467167142cf9e48d96f7b54001b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679838909477 2023-03-26 13:55:12,610 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 62f5cd7dc2354cde9befe212ed428027, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679838911653 2023-03-26 13:55:12,628 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a9871eaedebb91319d17cf11612be1e0#C#compaction#363 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:12,631 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303261c1a18e74cce48f086ee32e563c590fd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303261c1a18e74cce48f086ee32e563c590fd 2023-03-26 13:55:12,647 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/8980a7aa5ebc4dbd8d38d1d2dfca2b76 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/8980a7aa5ebc4dbd8d38d1d2dfca2b76 2023-03-26 13:55:12,652 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9871eaedebb91319d17cf11612be1e0/A of a9871eaedebb91319d17cf11612be1e0 into 8980a7aa5ebc4dbd8d38d1d2dfca2b76(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:12,652 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:12,652 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., storeName=a9871eaedebb91319d17cf11612be1e0/A, priority=13, startTime=1679838912151; duration=0sec 2023-03-26 13:55:12,652 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:12,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838972677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838972678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838972678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53804 deadline: 1679838972865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1679838972876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838972987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838972987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:12,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:12,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838972988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,041 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/5ee591eb493f42458f367162d572f9f8 2023-03-26 13:55:13,042 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/12317b30bb5b4190a3c16c0678f07381 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/12317b30bb5b4190a3c16c0678f07381 2023-03-26 13:55:13,047 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9871eaedebb91319d17cf11612be1e0/C of a9871eaedebb91319d17cf11612be1e0 into 12317b30bb5b4190a3c16c0678f07381(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:13,047 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:13,048 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., storeName=a9871eaedebb91319d17cf11612be1e0/C, priority=13, startTime=1679838912151; duration=0sec 2023-03-26 13:55:13,048 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:13,451 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/12708e34f8744bfe94f52e8e2cc412d9 2023-03-26 13:55:13,468 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/2955b3869afb48ae87bc6b0020fec528 2023-03-26 13:55:13,471 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/5ee591eb493f42458f367162d572f9f8 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5ee591eb493f42458f367162d572f9f8 2023-03-26 13:55:13,474 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5ee591eb493f42458f367162d572f9f8, entries=150, sequenceid=97, filesize=25.1 K 2023-03-26 13:55:13,475 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/12708e34f8744bfe94f52e8e2cc412d9 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/12708e34f8744bfe94f52e8e2cc412d9 2023-03-26 13:55:13,479 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/12708e34f8744bfe94f52e8e2cc412d9, entries=150, sequenceid=97, filesize=11.6 K 2023-03-26 13:55:13,479 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/2955b3869afb48ae87bc6b0020fec528 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/2955b3869afb48ae87bc6b0020fec528 2023-03-26 13:55:13,482 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/2955b3869afb48ae87bc6b0020fec528, entries=150, sequenceid=97, filesize=11.6 K 2023-03-26 13:55:13,482 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for a9871eaedebb91319d17cf11612be1e0 in 1270ms, sequenceid=97, compaction requested=false 2023-03-26 13:55:13,483 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:13,483 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:13,483 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:13,483 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:13,483 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,483 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:13,483 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:13,484 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,484 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:13,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:13,484 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,484 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:13,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:13,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4803ed8[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:13,486 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:13,486 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,487 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,487 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:13,487 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:13,487 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:13,487 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,487 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:13,488 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:13,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:13,489 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:13,489 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:13,489 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:13,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:13,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:13,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:13,490 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:13,490 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:13,490 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:13,490 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:13,490 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:13,493 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,493 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,493 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:13,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:13,494 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:13,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:13,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,495 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:13,495 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 13:55:13,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:13,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:13,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:13,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:13,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:13,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:13,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:13,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,498 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:13,498 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:13,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:13,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:13,498 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:13,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:13,498 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,498 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,498 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:13,498 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,498 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:13,498 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,498 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,498 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:13,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:13,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:13,498 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:13,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:13,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838973524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:13,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838973525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:13,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838973528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,599 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:13,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:13,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:13,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:13,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:13,603 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:13,603 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:13,603 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:13,603 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:13,604 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,604 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,605 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:13,605 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:13,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:13,606 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,606 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:13,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:13,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:13,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:13,607 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:13,607 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:13,607 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:13,608 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:13,608 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:13,608 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:13,608 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:13,608 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:13,608 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:13,608 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:13,608 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:13,608 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,608 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:13,608 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:13,609 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,609 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:13,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:13,610 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,610 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:13,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:13,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:13,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:13,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:13,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3465b24f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:13,611 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:13,611 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,612 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:13,612 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:13,612 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:13,613 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,613 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:13,615 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,615 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:13,615 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:13,615 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:13,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:13,618 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:13,618 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:13,618 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:13,618 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:13,618 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:13,619 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,619 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:13,619 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:13,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:13,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:13,620 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,621 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:13,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,623 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:13,623 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:13,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:13,623 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:13,623 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,623 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,623 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:13,624 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,624 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,624 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,624 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:13,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:13,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:13,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:13,625 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:13,625 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:13,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:13,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838973633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:13,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838973633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:13,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838973633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,725 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:13,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:13,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:13,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:13,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:13,732 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:13,732 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:13,732 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:13,732 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:13,732 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,732 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,734 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:13,734 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:13,735 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,735 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:13,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:13,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:13,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:13,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:13,736 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:13,736 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:13,737 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:13,738 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:13,738 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:13,738 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:13,738 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:13,738 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:13,738 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:13,738 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:13,739 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:13,739 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,739 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:13,739 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:13,745 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,745 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:13,745 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,746 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:13,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:13,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:13,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:13,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@899bd2c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:13,747 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:13,747 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,748 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,748 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,748 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:13,748 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:13,748 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:13,748 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:13,750 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,750 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:13,750 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:13,750 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:13,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:13,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:13,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:13,752 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:13,752 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:13,752 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:13,752 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:13,752 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:13,754 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,754 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,754 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:13,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:13,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:13,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:13,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:13,755 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:13,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:13,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,756 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:13,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,761 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:13,761 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:13,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:13,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:13,762 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,762 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,762 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,762 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,762 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,762 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,762 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,763 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:13,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:13,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:13,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:13,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:13,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:13,763 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:13,763 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:13,844 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:13,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838973840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,849 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:13,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838973846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:13,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838973847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,863 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:13,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:13,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:13,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:13,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:13,866 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:13,866 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:13,866 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:13,866 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:13,867 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,867 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,868 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:13,868 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:13,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:13,868 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,869 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:13,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:13,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,869 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:13,869 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:13,869 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:13,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:13,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:13,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:13,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:13,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:13,870 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:13,870 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:13,870 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:13,870 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:13,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:13,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:13,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:13,870 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:13,871 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,872 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:13,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:13,872 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,872 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:13,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:13,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:13,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:13,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:13,873 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:13,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b9ea1ef[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:13,873 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,874 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,875 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:13,875 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:13,875 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,875 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:13,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:13,875 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:13,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53804 deadline: 1679838973872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,877 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,878 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:13,878 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:13,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:13,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:13,878 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:13,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:13,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:13,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:13,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:13,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:13,880 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:13,880 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:13,880 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:13,880 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:13,880 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:13,882 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,882 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,882 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:13,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:13,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:13,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,882 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:13,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:13,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:13,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:13,884 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:13,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:13,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,886 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:13,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:13,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:13,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:13,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:13,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:13,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:13,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:13,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,887 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:13,887 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:13,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:13,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:13,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:13,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:13,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1679838973886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,908 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303264036d9c024db4199af987d9a0def7b0e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303264036d9c024db4199af987d9a0def7b0e 2023-03-26 13:55:13,913 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=112, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/8ff872b31fa74f24b3a38176c6162b1a 2023-03-26 13:55:13,927 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/36225ca64c274c45b13c3ff75d33f640 2023-03-26 13:55:13,987 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:13,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:13,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:13,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:13,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:13,994 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:13,994 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:13,994 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:13,994 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:13,994 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,995 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,996 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:13,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:13,996 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:13,996 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,996 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:13,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:13,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:13,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:13,997 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:13,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:13,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:13,997 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:13,997 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:13,997 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:13,998 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:13,998 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:13,998 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:13,998 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:13,998 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:13,998 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:13,998 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:13,999 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:13,999 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:13,999 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:13,999 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:14,001 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,001 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:14,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:14,002 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,002 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:14,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:14,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:14,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:14,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:14,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:14,003 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:14,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@eed28ba[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:14,003 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,005 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,006 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:14,006 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,006 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:14,006 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:14,006 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:14,007 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:14,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:14,007 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:14,007 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:14,007 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:14,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:14,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:14,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:14,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:14,009 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:14,010 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:14,010 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:14,010 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:14,010 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:14,011 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,011 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:14,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,011 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:14,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:14,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:14,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:14,012 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:14,013 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:14,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:14,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,017 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,017 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:14,017 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:14,017 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,017 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:14,017 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:14,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:14,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:14,017 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:14,017 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:14,017 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,017 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,017 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:14,017 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:14,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:14,117 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:14,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:14,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:14,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:14,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:14,124 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:14,124 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:14,124 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:14,124 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:14,124 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,125 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:14,126 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:14,126 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:14,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:14,126 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:14,127 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:14,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:14,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:14,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:14,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:14,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:14,132 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:14,132 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:14,132 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:14,133 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:14,133 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:14,133 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:14,133 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:14,133 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:14,133 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:14,133 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:14,133 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:14,133 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,134 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:14,134 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:14,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:14,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:14,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:14,135 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:14,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:14,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:14,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:14,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:14,137 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:14,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@170314e2[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:14,137 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,139 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,139 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:14,139 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:14,139 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:14,143 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:14,144 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:14,144 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:14,144 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:14,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:14,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:14,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:14,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:14,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:14,146 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:14,146 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:14,146 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:14,146 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:14,146 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:14,147 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,147 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,147 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:14,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:14,148 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:14,149 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:14,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:14,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:14,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838974146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:14,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838974150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,151 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:14,151 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:14,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:14,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:14,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:14,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:14,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:14,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:14,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,152 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:14,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:14,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:14,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:14,153 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:14,153 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:14,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:14,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838974152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,159 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,159 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,253 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:14,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:14,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:14,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:14,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:14,261 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:14,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:14,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:14,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:14,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:14,262 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:14,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:14,263 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,263 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:14,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:14,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:14,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:14,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:14,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:14,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:14,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:14,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:14,272 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:14,272 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:14,272 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:14,272 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:14,272 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:14,272 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:14,272 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:14,272 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:14,272 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,272 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:14,272 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:14,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,276 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:14,276 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,276 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:14,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:14,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:14,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:14,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:14,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52b6bbf3[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:14,278 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:14,278 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,282 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:14,282 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,282 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:14,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:14,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:14,283 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:14,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:14,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:14,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:14,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:14,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:14,288 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:14,288 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:14,288 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:14,288 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:14,288 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:14,290 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,290 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:14,290 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:14,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:14,290 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:14,292 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:14,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:14,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:14,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:14,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:14,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:14,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:14,295 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:14,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:14,296 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:14,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:14,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:14,296 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:14,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:14,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:14,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:14,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=112 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/9a9d5468e7954a13b61a7b1fdd46e481 2023-03-26 13:55:14,349 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/8ff872b31fa74f24b3a38176c6162b1a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/8ff872b31fa74f24b3a38176c6162b1a 2023-03-26 13:55:14,352 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/8ff872b31fa74f24b3a38176c6162b1a, entries=200, sequenceid=112, filesize=31.9 K 2023-03-26 13:55:14,353 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/36225ca64c274c45b13c3ff75d33f640 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/36225ca64c274c45b13c3ff75d33f640 2023-03-26 13:55:14,356 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/36225ca64c274c45b13c3ff75d33f640, entries=150, sequenceid=112, filesize=11.6 K 2023-03-26 13:55:14,357 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/9a9d5468e7954a13b61a7b1fdd46e481 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/9a9d5468e7954a13b61a7b1fdd46e481 2023-03-26 13:55:14,361 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/9a9d5468e7954a13b61a7b1fdd46e481, entries=150, sequenceid=112, filesize=11.6 K 2023-03-26 13:55:14,362 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a9871eaedebb91319d17cf11612be1e0 in 867ms, sequenceid=112, compaction requested=true 2023-03-26 13:55:14,362 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:14,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:14,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:14,362 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:14,362 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:14,362 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:14,363 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84281 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:14,363 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a9871eaedebb91319d17cf11612be1e0/A is initiating minor compaction (all files) 2023-03-26 13:55:14,363 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:14,363 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a9871eaedebb91319d17cf11612be1e0/A in TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:14,363 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a9871eaedebb91319d17cf11612be1e0/B is initiating minor compaction (all files) 2023-03-26 13:55:14,363 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/8980a7aa5ebc4dbd8d38d1d2dfca2b76, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5ee591eb493f42458f367162d572f9f8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/8ff872b31fa74f24b3a38176c6162b1a] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp, totalSize=82.3 K 2023-03-26 13:55:14,363 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9871eaedebb91319d17cf11612be1e0/B in TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:14,364 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/dcfb160bfa4e47d58fb8acdbfc33eb86, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/12708e34f8744bfe94f52e8e2cc412d9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/36225ca64c274c45b13c3ff75d33f640] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp, totalSize=34.9 K 2023-03-26 13:55:14,364 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 8980a7aa5ebc4dbd8d38d1d2dfca2b76, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679838911653 2023-03-26 13:55:14,364 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting dcfb160bfa4e47d58fb8acdbfc33eb86, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679838911653 2023-03-26 13:55:14,364 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 5ee591eb493f42458f367162d572f9f8, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679838911730 2023-03-26 13:55:14,365 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 12708e34f8744bfe94f52e8e2cc412d9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679838911730 2023-03-26 13:55:14,365 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 8ff872b31fa74f24b3a38176c6162b1a, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1679838912344 2023-03-26 13:55:14,366 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 36225ca64c274c45b13c3ff75d33f640, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1679838912344 2023-03-26 13:55:14,377 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a9871eaedebb91319d17cf11612be1e0#B#compaction#369 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:14,396 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:14,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:14,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:14,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:14,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:14,418 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:14,418 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:14,418 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:14,418 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:14,418 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,418 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:14,419 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a9871eaedebb91319d17cf11612be1e0#A#compaction#370 average throughput is 0.56 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:14,419 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:14,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:14,419 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:14,420 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,420 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:14,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:14,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:14,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:14,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:14,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:14,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:14,423 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:14,423 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:14,423 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:14,425 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/cdbdd69f04e14654a50892575667f086 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/cdbdd69f04e14654a50892575667f086 2023-03-26 13:55:14,433 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9871eaedebb91319d17cf11612be1e0/B of a9871eaedebb91319d17cf11612be1e0 into cdbdd69f04e14654a50892575667f086(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:14,433 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:14,433 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., storeName=a9871eaedebb91319d17cf11612be1e0/B, priority=13, startTime=1679838914362; duration=0sec 2023-03-26 13:55:14,433 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:14,433 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:14,435 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:14,435 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:14,435 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:14,435 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:14,435 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-26 13:55:14,435 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:14,435 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a9871eaedebb91319d17cf11612be1e0/C is initiating minor compaction (all files) 2023-03-26 13:55:14,435 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:14,436 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9871eaedebb91319d17cf11612be1e0/C in TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:14,436 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:14,436 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/12317b30bb5b4190a3c16c0678f07381, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/2955b3869afb48ae87bc6b0020fec528, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/9a9d5468e7954a13b61a7b1fdd46e481] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp, totalSize=34.9 K 2023-03-26 13:55:14,436 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:14,436 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:14,436 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:14,436 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:14,436 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 12317b30bb5b4190a3c16c0678f07381, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679838911653 2023-03-26 13:55:14,437 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 2955b3869afb48ae87bc6b0020fec528, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1679838911730 2023-03-26 13:55:14,437 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 9a9d5468e7954a13b61a7b1fdd46e481, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1679838912344 2023-03-26 13:55:14,455 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a9871eaedebb91319d17cf11612be1e0#C#compaction#371 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:14,476 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/f378b8b6ea604720b75114399539156a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/f378b8b6ea604720b75114399539156a 2023-03-26 13:55:14,480 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9871eaedebb91319d17cf11612be1e0/C of a9871eaedebb91319d17cf11612be1e0 into f378b8b6ea604720b75114399539156a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:14,480 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:14,480 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., storeName=a9871eaedebb91319d17cf11612be1e0/C, priority=13, startTime=1679838914362; duration=0sec 2023-03-26 13:55:14,480 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:14,489 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326e71d95465d984d5ebdb6f2012869f3b1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e71d95465d984d5ebdb6f2012869f3b1 2023-03-26 13:55:14,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:14,657 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:14,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:14,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838974668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:14,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838974669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:14,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838974670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:14,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838974771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:14,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838974774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:14,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838974774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,894 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=133, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/01b7d924b1fb4de2b340be18c61d73f5 2023-03-26 13:55:14,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:14,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838974976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:14,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838974978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:14,982 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:14,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838974978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,275 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/f21b3ba02e0c4d12a7b38cc98ff3b7d8 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/f21b3ba02e0c4d12a7b38cc98ff3b7d8 2023-03-26 13:55:15,279 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9871eaedebb91319d17cf11612be1e0/A of a9871eaedebb91319d17cf11612be1e0 into f21b3ba02e0c4d12a7b38cc98ff3b7d8(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:15,279 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:15,279 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., storeName=a9871eaedebb91319d17cf11612be1e0/A, priority=13, startTime=1679838914362; duration=0sec 2023-03-26 13:55:15,279 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:15,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:15,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838975281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:15,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838975283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:15,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838975285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,303 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/ec196239ef104244aacfe16f0945cfd4 2023-03-26 13:55:15,313 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/1e823dee8e8b460db53be47f09b4e16c 2023-03-26 13:55:15,316 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/01b7d924b1fb4de2b340be18c61d73f5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/01b7d924b1fb4de2b340be18c61d73f5 2023-03-26 13:55:15,319 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/01b7d924b1fb4de2b340be18c61d73f5, entries=150, sequenceid=133, filesize=25.3 K 2023-03-26 13:55:15,320 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/ec196239ef104244aacfe16f0945cfd4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/ec196239ef104244aacfe16f0945cfd4 2023-03-26 13:55:15,323 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/ec196239ef104244aacfe16f0945cfd4, entries=150, sequenceid=133, filesize=11.8 K 2023-03-26 13:55:15,323 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/1e823dee8e8b460db53be47f09b4e16c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/1e823dee8e8b460db53be47f09b4e16c 2023-03-26 13:55:15,326 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/1e823dee8e8b460db53be47f09b4e16c, entries=150, sequenceid=133, filesize=11.8 K 2023-03-26 13:55:15,327 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a9871eaedebb91319d17cf11612be1e0 in 892ms, sequenceid=133, compaction requested=false 2023-03-26 13:55:15,327 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:15,327 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:15,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:15,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:15,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:15,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:15,328 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:15,329 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:15,329 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:15,330 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:15,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:15,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:15,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:15,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cd9e993[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:15,331 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:15,331 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:15,332 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:15,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:15,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:15,333 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:15,333 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:15,333 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:15,333 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,333 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:15,335 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,335 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:15,335 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:15,335 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:15,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:15,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:15,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:15,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:15,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:15,337 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:15,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:15,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:15,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:15,337 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:15,339 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:15,339 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:15,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:15,339 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:15,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:15,339 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:15,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:15,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:15,342 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:15,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,344 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,344 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:15,344 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:15,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:15,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:15,344 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:15,344 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:15,344 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,344 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:15,344 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:15,344 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:15,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:15,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:15,344 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:15,344 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:15,344 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:15,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:15,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:15,444 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:15,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:15,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:15,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:15,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:15,447 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:15,447 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:15,447 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:15,447 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:15,448 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:15,448 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:15,449 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:15,449 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:15,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:15,450 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,450 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:15,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:15,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:15,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:15,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:15,450 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:15,450 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:15,450 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:15,451 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:15,451 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:15,451 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:15,451 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:15,451 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 13:55:15,451 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:15,451 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:15,451 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:15,451 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:15,451 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:15,452 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:15,463 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303263b58116b94444022bc6ef85e0d70208b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263b58116b94444022bc6ef85e0d70208b 2023-03-26 13:55:15,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:15,799 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:15,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:15,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838975827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:15,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838975831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:15,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838975831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,868 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=147, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/0437d49d2c2e4bc09c378c1c0ceb9646 2023-03-26 13:55:15,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:15,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53804 deadline: 1679838975893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,898 DEBUG [Thread-2669] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., hostname=jenkins-hbase4.apache.org,42517,1679838814721, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:55:15,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:15,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1679838975903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,905 DEBUG [Thread-2677] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4161 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., hostname=jenkins-hbase4.apache.org,42517,1679838814721, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:55:15,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:15,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838975934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:15,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838975934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:15,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:15,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838975937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:16,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838976139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:16,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838976139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:16,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838976142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,278 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/cbe17c6cf273476cb8a4252beae59b99 2023-03-26 13:55:16,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:16,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838976443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:16,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838976444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,450 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:16,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838976447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,690 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/1853a75ba65d42de8604d30f0cd5b31b 2023-03-26 13:55:16,693 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/0437d49d2c2e4bc09c378c1c0ceb9646 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/0437d49d2c2e4bc09c378c1c0ceb9646 2023-03-26 13:55:16,696 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/0437d49d2c2e4bc09c378c1c0ceb9646, entries=150, sequenceid=147, filesize=25.3 K 2023-03-26 13:55:16,697 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/cbe17c6cf273476cb8a4252beae59b99 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/cbe17c6cf273476cb8a4252beae59b99 2023-03-26 13:55:16,700 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/cbe17c6cf273476cb8a4252beae59b99, entries=150, sequenceid=147, filesize=11.8 K 2023-03-26 13:55:16,700 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/1853a75ba65d42de8604d30f0cd5b31b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/1853a75ba65d42de8604d30f0cd5b31b 2023-03-26 13:55:16,703 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/1853a75ba65d42de8604d30f0cd5b31b, entries=150, sequenceid=147, filesize=11.8 K 2023-03-26 13:55:16,712 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a9871eaedebb91319d17cf11612be1e0 in 1261ms, sequenceid=147, compaction requested=true 2023-03-26 13:55:16,712 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:16,712 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:16,712 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:16,712 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:16,712 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,712 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:16,712 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:16,714 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,714 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:16,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:16,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:16,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:16,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:16,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:16,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b16bc4a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:16,716 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:16,716 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:16,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:16,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:16,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:16,718 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:16,718 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:16,718 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:16,718 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,718 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:16,719 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:16,719 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:16,719 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,719 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:16,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:16,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:16,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:16,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:16,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:16,721 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:16,721 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:16,721 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:16,721 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:16,721 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:16,727 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:16,727 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:16,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:16,727 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:16,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:16,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:16,728 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:16,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:16,729 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:16,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:16,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,730 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:16,731 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:16,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:16,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:16,731 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:16,731 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,731 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:16,731 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:16,731 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,731 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:16,731 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:16,731 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:16,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:16,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:16,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:16,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:16,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:16,732 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:16,732 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:16,832 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:16,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:16,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:16,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:16,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:16,835 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:16,835 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:16,835 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:16,835 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:16,836 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:16,836 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:16,837 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,837 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:16,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:16,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:16,837 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,837 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:16,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:16,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:16,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:16,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:16,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:16,838 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:16,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:16,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:16,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:16,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:16,839 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:16,839 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:16,839 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-26 13:55:16,839 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:16,839 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:16,839 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:16,839 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:16,840 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:16,840 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:16,850 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303260b2d93648354458596fe494f8f9f2df8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260b2d93648354458596fe494f8f9f2df8 2023-03-26 13:55:16,855 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=167, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/12fdc39eff514e01bf482ee463d5b692 2023-03-26 13:55:16,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:16,953 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:16,970 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:16,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838976966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,975 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:16,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838976969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:16,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:16,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838976970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:17,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838977071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:17,086 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:17,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838977076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838977076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,266 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/515e6f451db846d8bb73ceb384c7b202 2023-03-26 13:55:17,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:17,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838977276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:17,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838977287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:17,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838977288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:17,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838977581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:17,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838977590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:17,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838977593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,675 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/54248848bbac4caa8f61092bc8b4ecf7 2023-03-26 13:55:17,678 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/12fdc39eff514e01bf482ee463d5b692 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/12fdc39eff514e01bf482ee463d5b692 2023-03-26 13:55:17,681 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/12fdc39eff514e01bf482ee463d5b692, entries=150, sequenceid=167, filesize=25.3 K 2023-03-26 13:55:17,683 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/515e6f451db846d8bb73ceb384c7b202 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/515e6f451db846d8bb73ceb384c7b202 2023-03-26 13:55:17,686 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/515e6f451db846d8bb73ceb384c7b202, entries=150, sequenceid=167, filesize=11.8 K 2023-03-26 13:55:17,686 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/54248848bbac4caa8f61092bc8b4ecf7 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/54248848bbac4caa8f61092bc8b4ecf7 2023-03-26 13:55:17,690 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/54248848bbac4caa8f61092bc8b4ecf7, entries=150, sequenceid=167, filesize=11.8 K 2023-03-26 13:55:17,690 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for a9871eaedebb91319d17cf11612be1e0 in 851ms, sequenceid=167, compaction requested=true 2023-03-26 13:55:17,691 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:17,691 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:17,691 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:17,691 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:17,691 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,691 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:17,691 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:17,692 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:17,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:17,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:17,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:17,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:17,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44bd15ce[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:17,694 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:17,694 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:17,696 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:17,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:17,696 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:17,696 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:17,696 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:17,696 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,696 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:17,697 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,697 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:17,697 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:17,697 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:17,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:17,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:17,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:17,699 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:17,699 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:17,700 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:17,700 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:17,700 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:17,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:17,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:17,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:17,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:17,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:17,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:17,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:17,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:17,702 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:17,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:17,703 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:17,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:17,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,705 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:17,705 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,705 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:17,705 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:17,705 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:17,705 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:17,705 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,705 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:17,705 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:17,705 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:17,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:17,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:17,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:17,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:17,707 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:17,707 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:17,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:17,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:17,807 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:17,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:17,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:17,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:17,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:17,810 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:17,810 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:17,810 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:17,810 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:17,811 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:17,811 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:17,813 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,813 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:17,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:17,814 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:17,814 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:17,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:17,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:17,815 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:17,815 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:17,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:17,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:17,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:17,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:17,815 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:17,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:17,815 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:17,815 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-26 13:55:17,815 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:17,815 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:17,816 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:17,816 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:17,816 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:17,816 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:17,828 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326f290fe9766ea40dfa610a214bbdc2cc3 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f290fe9766ea40dfa610a214bbdc2cc3 2023-03-26 13:55:18,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:18,092 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:18,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:18,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838978147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:18,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838978148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:18,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838978152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,246 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=180, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/5989223eab014cbdb1a33966a307f891 2023-03-26 13:55:18,259 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/f3e527038832454dab9be1b9d4b0a85c 2023-03-26 13:55:18,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:18,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838978259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:18,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838978259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:18,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838978261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,269 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=180 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/28959ac0406f47a5bc2e7af6274c3433 2023-03-26 13:55:18,274 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/5989223eab014cbdb1a33966a307f891 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5989223eab014cbdb1a33966a307f891 2023-03-26 13:55:18,277 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5989223eab014cbdb1a33966a307f891, entries=150, sequenceid=180, filesize=25.3 K 2023-03-26 13:55:18,277 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/f3e527038832454dab9be1b9d4b0a85c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/f3e527038832454dab9be1b9d4b0a85c 2023-03-26 13:55:18,281 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/f3e527038832454dab9be1b9d4b0a85c, entries=150, sequenceid=180, filesize=11.8 K 2023-03-26 13:55:18,282 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/28959ac0406f47a5bc2e7af6274c3433 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/28959ac0406f47a5bc2e7af6274c3433 2023-03-26 13:55:18,285 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/28959ac0406f47a5bc2e7af6274c3433, entries=150, sequenceid=180, filesize=11.8 K 2023-03-26 13:55:18,286 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for a9871eaedebb91319d17cf11612be1e0 in 471ms, sequenceid=180, compaction requested=true 2023-03-26 13:55:18,286 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:18,286 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:18,286 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:18,286 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:18,286 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,286 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:18,286 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:18,288 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:18,288 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:18,288 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:18,288 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:18,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:18,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23b35789[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:18,290 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:18,290 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:18,291 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:18,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:18,291 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:18,291 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,291 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:18,292 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:18,292 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:18,293 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:18,293 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:18,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:18,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:18,295 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:18,295 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:18,295 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:18,295 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:18,295 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:18,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:18,298 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:18,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:18,298 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:18,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:18,298 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,300 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:18,301 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,301 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:18,301 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:18,302 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,302 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:18,302 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:18,302 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:18,302 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:18,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:18,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:18,302 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:18,302 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:18,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:18,302 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:18,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:18,302 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:18,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:18,402 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:18,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:18,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:18,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:18,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:18,405 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:18,406 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:18,406 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:18,406 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:18,406 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:18,406 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:18,407 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:18,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:18,408 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:18,408 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:18,408 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:18,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:18,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:18,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:18,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:18,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:18,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:18,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:18,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:18,409 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:18,409 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:18,409 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-26 13:55:18,409 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:18,409 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:18,409 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:18,409 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:18,409 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:18,409 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:18,422 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303268284d7e73075478c9c9b690ef8422f58 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303268284d7e73075478c9c9b690ef8422f58 2023-03-26 13:55:18,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:18,471 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:18,508 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:18,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838978494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,522 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:18,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:18,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838978509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838978509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:18,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838978609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:18,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838978624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:18,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838978624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:18,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838978816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,829 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=199, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/3b10b87c8599410a846f708a8fb58650 2023-03-26 13:55:18,843 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:18,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838978838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:18,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838978838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:18,846 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/c0c3d832cc124f5689c21c97010ae868 2023-03-26 13:55:19,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:19,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838979125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:19,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838979146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:19,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838979147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,255 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/3e3ff27aff484d0494be6293e74c52ca 2023-03-26 13:55:19,259 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/3b10b87c8599410a846f708a8fb58650 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/3b10b87c8599410a846f708a8fb58650 2023-03-26 13:55:19,263 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/3b10b87c8599410a846f708a8fb58650, entries=150, sequenceid=199, filesize=25.3 K 2023-03-26 13:55:19,264 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/c0c3d832cc124f5689c21c97010ae868 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/c0c3d832cc124f5689c21c97010ae868 2023-03-26 13:55:19,273 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/c0c3d832cc124f5689c21c97010ae868, entries=150, sequenceid=199, filesize=11.8 K 2023-03-26 13:55:19,274 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/3e3ff27aff484d0494be6293e74c52ca as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/3e3ff27aff484d0494be6293e74c52ca 2023-03-26 13:55:19,279 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/3e3ff27aff484d0494be6293e74c52ca, entries=150, sequenceid=199, filesize=11.8 K 2023-03-26 13:55:19,280 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for a9871eaedebb91319d17cf11612be1e0 in 871ms, sequenceid=199, compaction requested=true 2023-03-26 13:55:19,280 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:19,280 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:19,281 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:19,281 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:19,281 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,281 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:19,281 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:19,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:19,282 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:19,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:19,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:19,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:19,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:19,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40c5b71d[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:19,285 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:19,285 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:19,288 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:19,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:19,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:19,288 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:19,288 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:19,288 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:19,288 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,288 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:19,291 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,291 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:19,291 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:19,291 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:19,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:19,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:19,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:19,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:19,295 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:19,295 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:19,295 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:19,295 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:19,295 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:19,298 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:19,298 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:19,298 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:19,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:19,298 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:19,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:19,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:19,301 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,306 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:19,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:19,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:19,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:19,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:19,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:19,306 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:19,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:19,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:19,306 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:19,307 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,307 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:19,308 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:19,308 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:19,308 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:19,309 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,309 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:19,309 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:19,309 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:19,409 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:19,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:19,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:19,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:19,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:19,413 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:19,413 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:19,413 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:19,413 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:19,413 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:19,414 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:19,415 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:19,415 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:19,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:19,417 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,417 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:19,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:19,418 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:19,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:19,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:19,418 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:19,418 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:19,418 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:19,418 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:19,418 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:19,418 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:19,418 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:19,419 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-26 13:55:19,419 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:19,419 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:19,419 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:19,419 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:19,419 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:19,419 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:19,464 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032623ca1082e8974133a086e1e7b0e67d41 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032623ca1082e8974133a086e1e7b0e67d41 2023-03-26 13:55:19,487 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=213, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/29c97c21827f4d2ea89a4a3d30c33460 2023-03-26 13:55:19,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:19,631 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:19,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:19,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838979692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,700 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:19,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838979694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:19,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838979695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:19,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838979796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:19,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838979801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:19,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838979804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,908 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/82d03ddcd7dc45de8e5b044818ad6f44 2023-03-26 13:55:19,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:19,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53804 deadline: 1679838979912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,915 DEBUG [Thread-2669] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., hostname=jenkins-hbase4.apache.org,42517,1679838814721, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:55:19,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:19,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1679838979931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:19,940 DEBUG [Thread-2677] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8196 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., hostname=jenkins-hbase4.apache.org,42517,1679838814721, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:55:20,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:20,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838980003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:20,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838980007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:20,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53818 deadline: 1679838980012, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,319 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/3b2876f4916f45f7a1c5861894903913 2023-03-26 13:55:20,322 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/29c97c21827f4d2ea89a4a3d30c33460 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/29c97c21827f4d2ea89a4a3d30c33460 2023-03-26 13:55:20,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:20,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53826 deadline: 1679838980319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,326 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/29c97c21827f4d2ea89a4a3d30c33460, entries=150, sequenceid=213, filesize=25.3 K 2023-03-26 13:55:20,326 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/82d03ddcd7dc45de8e5b044818ad6f44 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/82d03ddcd7dc45de8e5b044818ad6f44 2023-03-26 13:55:20,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:20,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53846 deadline: 1679838980322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,329 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/82d03ddcd7dc45de8e5b044818ad6f44, entries=150, sequenceid=213, filesize=11.8 K 2023-03-26 13:55:20,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:20,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838980324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,330 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/3b2876f4916f45f7a1c5861894903913 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/3b2876f4916f45f7a1c5861894903913 2023-03-26 13:55:20,333 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/3b2876f4916f45f7a1c5861894903913, entries=150, sequenceid=213, filesize=11.8 K 2023-03-26 13:55:20,334 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for a9871eaedebb91319d17cf11612be1e0 in 916ms, sequenceid=213, compaction requested=true 2023-03-26 13:55:20,334 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:20,334 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:20,334 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:20,334 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:20,334 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,334 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:20,334 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:20,340 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:20,340 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:20,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:20,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ecfb7a3[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:20,342 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:20,342 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:20,343 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:20,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:20,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:20,343 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:20,343 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,343 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:20,343 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:20,343 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:20,345 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,345 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:20,345 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:20,345 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:20,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:20,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:20,347 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:20,347 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:20,347 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:20,347 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:20,347 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:20,349 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:20,350 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:20,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,352 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:20,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,354 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:20,354 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:20,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:20,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:20,356 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:20,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:20,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:20,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:20,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:20,356 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:20,356 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,356 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:20,356 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:20,356 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:20,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:20,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:20,356 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,356 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:20,356 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:20,357 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:20,357 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:20,457 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:20,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:20,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:20,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:20,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:20,460 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:20,460 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:20,460 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:20,460 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:20,460 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:20,460 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:20,461 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:20,461 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:20,462 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,462 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:20,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:20,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:20,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:20,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:20,463 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:20,463 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:20,463 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:20,463 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:20,463 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:20,463 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:20,463 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:20,463 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-26 13:55:20,464 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:20,464 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:20,464 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:20,464 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:20,464 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:20,464 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:20,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:20,829 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:20,862 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:20,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838980854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:20,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838980859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:20,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838980860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,876 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303260adfed3fa40a4d54b4416c82054ad710 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260adfed3fa40a4d54b4416c82054ad710 2023-03-26 13:55:20,881 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=231, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/e3a2b1e18471428c8f280078d383e45d 2023-03-26 13:55:20,894 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/c09d0d50f46941cd9f12f3779ef15e2e 2023-03-26 13:55:20,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:20,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838980963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:20,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838980970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:20,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:20,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838980970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:21,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838981176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,187 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:21,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838981181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:21,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838981182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,311 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/dc2d9621b6684c85b6f885b31b2ad9aa 2023-03-26 13:55:21,314 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/e3a2b1e18471428c8f280078d383e45d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/e3a2b1e18471428c8f280078d383e45d 2023-03-26 13:55:21,318 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/e3a2b1e18471428c8f280078d383e45d, entries=150, sequenceid=231, filesize=25.3 K 2023-03-26 13:55:21,318 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/c09d0d50f46941cd9f12f3779ef15e2e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/c09d0d50f46941cd9f12f3779ef15e2e 2023-03-26 13:55:21,321 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/c09d0d50f46941cd9f12f3779ef15e2e, entries=150, sequenceid=231, filesize=11.8 K 2023-03-26 13:55:21,322 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/dc2d9621b6684c85b6f885b31b2ad9aa as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/dc2d9621b6684c85b6f885b31b2ad9aa 2023-03-26 13:55:21,325 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/dc2d9621b6684c85b6f885b31b2ad9aa, entries=150, sequenceid=231, filesize=11.8 K 2023-03-26 13:55:21,325 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for a9871eaedebb91319d17cf11612be1e0 in 862ms, sequenceid=231, compaction requested=true 2023-03-26 13:55:21,325 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:21,325 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:21,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:21,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:21,326 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,326 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:21,326 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:21,327 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:21,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:21,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:21,335 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,335 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:21,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:21,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1270c6d1[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:21,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:21,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,338 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,338 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:21,338 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:21,338 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:21,338 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,338 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:21,340 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,340 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:21,340 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:21,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:21,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:21,340 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:21,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:21,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:21,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:21,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:21,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:21,343 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:21,343 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:21,343 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:21,343 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:21,343 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:21,344 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,344 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:21,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,344 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:21,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:21,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:21,344 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:21,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,346 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,347 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,347 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:21,347 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:21,347 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:21,348 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,348 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:21,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:21,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:21,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:21,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:21,348 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,348 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,348 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,348 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:21,348 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,348 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:21,448 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:21,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:21,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:21,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:21,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:21,451 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:21,451 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:21,451 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:21,451 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:21,452 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,452 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,453 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:21,453 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:21,454 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,454 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,454 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:21,455 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:21,455 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:21,455 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:21,455 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:21,455 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:21,455 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:21,455 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:21,455 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:21,455 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:21,455 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-26 13:55:21,455 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:21,456 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:21,456 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:21,456 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:21,456 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:21,456 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:21,473 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326022c9e96aae9499fab61eca14fa5071a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326022c9e96aae9499fab61eca14fa5071a 2023-03-26 13:55:21,481 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=245, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/b0ca98eb4c7440f29cc1ce72ec7ca083 2023-03-26 13:55:21,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:21,489 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:21,502 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/0ca0c294676c49969508c2eae85999ae 2023-03-26 13:55:21,528 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/be4253f0533a4d3fb8fbc154e9100cb4 2023-03-26 13:55:21,532 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/b0ca98eb4c7440f29cc1ce72ec7ca083 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/b0ca98eb4c7440f29cc1ce72ec7ca083 2023-03-26 13:55:21,535 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/b0ca98eb4c7440f29cc1ce72ec7ca083, entries=150, sequenceid=245, filesize=25.3 K 2023-03-26 13:55:21,536 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/0ca0c294676c49969508c2eae85999ae as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/0ca0c294676c49969508c2eae85999ae 2023-03-26 13:55:21,540 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/0ca0c294676c49969508c2eae85999ae, entries=150, sequenceid=245, filesize=11.8 K 2023-03-26 13:55:21,541 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/be4253f0533a4d3fb8fbc154e9100cb4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/be4253f0533a4d3fb8fbc154e9100cb4 2023-03-26 13:55:21,544 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/be4253f0533a4d3fb8fbc154e9100cb4, entries=150, sequenceid=245, filesize=11.8 K 2023-03-26 13:55:21,545 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=67.09 KB/68700 for a9871eaedebb91319d17cf11612be1e0 in 90ms, sequenceid=245, compaction requested=true 2023-03-26 13:55:21,545 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:21,545 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:21,545 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:21,546 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:21,546 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,546 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:21,546 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:21,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:21,547 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-26 13:55:21,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:21,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:21,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:21,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:21,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:21,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:21,548 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:21,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:21,549 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:21,549 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,549 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:21,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:21,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:21,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:21,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bc6c409[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:21,551 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:21,551 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,552 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,553 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,553 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:21,553 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:21,553 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:21,553 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:21,554 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,555 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:21,555 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:21,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:21,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:21,555 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:21,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:21,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:21,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:21,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:21,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:21,557 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:21,557 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:21,557 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:21,557 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:21,557 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:21,559 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,559 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:21,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:21,559 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:21,559 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:21,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:21,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:21,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:21,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,561 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:21,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,567 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:21,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:21,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:21,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:21,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:21,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:21,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:21,568 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:21,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:21,568 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,568 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:21,568 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,568 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:21,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838981619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:21,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838981619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:21,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838981620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,668 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:21,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:21,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:21,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:21,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:21,672 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:21,672 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:21,672 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:21,672 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:21,672 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,672 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,674 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:21,674 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:21,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:21,674 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,674 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:21,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:21,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:21,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:21,675 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:21,675 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:21,675 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:21,675 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:21,675 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:21,675 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:21,675 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:21,675 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:21,676 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:21,676 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:21,676 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:21,676 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,676 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:21,676 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:21,677 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,677 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,678 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:21,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:21,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4656e333[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:21,679 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:21,679 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,680 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:21,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:21,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:21,681 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,681 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:21,683 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:21,683 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:21,683 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:21,683 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:21,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:21,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:21,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:21,686 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:21,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:21,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:21,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:21,686 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:21,689 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,689 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,689 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:21,689 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:21,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:21,690 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:21,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:21,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:21,692 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:21,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:21,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:21,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:21,693 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,693 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,693 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:21,693 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:21,693 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:21,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:21,693 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:21,693 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,693 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:21,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838981738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:21,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838981738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:21,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838981738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,793 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:21,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:21,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:21,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:21,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:21,796 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:21,796 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:21,796 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:21,796 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:21,797 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,797 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,798 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:21,798 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:21,798 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,798 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:21,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:21,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:21,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:21,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:21,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:21,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:21,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:21,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:21,800 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:21,800 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:21,800 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:21,800 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:21,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:21,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:21,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:21,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:21,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,803 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:21,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:21,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:21,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56726f5[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:21,805 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:21,805 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,806 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,806 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:21,806 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:21,806 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:21,806 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,806 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:21,807 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:21,807 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:21,807 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,808 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:21,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:21,810 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:21,810 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:21,810 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:21,810 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:21,810 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:21,812 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,812 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:21,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:21,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,812 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:21,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:21,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:21,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:21,812 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,813 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,817 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,817 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,817 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:21,817 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,817 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:21,817 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:21,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:21,817 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:21,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:21,817 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,817 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,817 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:21,817 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:21,817 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:21,917 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:21,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:21,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:21,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:21,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:21,921 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:21,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:21,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:21,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:21,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,922 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:21,922 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:21,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:21,923 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,923 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:21,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:21,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:21,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:21,924 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:21,924 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:21,924 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:21,924 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:21,926 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:21,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:21,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:21,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11353ec6[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:21,927 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:21,927 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,928 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:21,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:21,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:21,929 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,929 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:21,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:21,931 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:21,931 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:21,931 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:21,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:21,933 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:21,933 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:21,933 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:21,933 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:21,933 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:21,934 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,934 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,934 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:21,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:21,935 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,936 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,938 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:21,938 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:21,938 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:21,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:21,938 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:21,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:21,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:21,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:21,938 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:21,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:21,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:21,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:21,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:21,939 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:21,939 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:21,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:21,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838981946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:21,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838981947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:21,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838981948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:21,962 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032624f092d785dc472a8a1173b351da3010 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032624f092d785dc472a8a1173b351da3010 2023-03-26 13:55:22,039 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:22,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:22,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:22,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:22,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:22,042 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:22,042 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:22,042 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:22,042 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:22,043 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,043 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,044 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:22,044 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:22,044 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,044 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:22,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:22,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:22,046 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:22,046 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:22,046 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:22,046 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:22,051 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,051 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,051 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,051 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:22,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cce4665[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:22,053 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:22,053 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,054 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:22,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:22,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:22,054 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,054 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:22,055 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,055 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:22,055 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:22,055 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:22,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:22,057 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:22,058 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:22,058 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:22,058 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:22,058 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:22,059 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,059 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,059 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:22,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:22,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:22,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,060 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,064 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,064 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:22,064 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,064 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:22,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:22,064 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:22,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:22,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:22,064 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,064 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,064 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:22,064 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:22,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:22,064 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:22,064 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,065 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,165 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:22,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:22,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:22,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:22,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:22,168 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:22,168 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:22,168 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:22,168 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:22,168 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,168 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,169 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:22,169 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:22,170 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,170 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:22,171 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:22,171 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:22,171 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:22,171 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:22,171 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:22,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:22,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:22,175 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:22,175 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:22,176 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:22,176 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:22,176 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:22,178 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,178 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,180 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,180 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:22,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7df063b4[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:22,182 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:22,182 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,183 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,183 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:22,183 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:22,183 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:22,183 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,183 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:22,185 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,185 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:22,185 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,185 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:22,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:22,188 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:22,188 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:22,188 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:22,188 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:22,188 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:22,191 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,191 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,191 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,192 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,193 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:22,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,195 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:22,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:22,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:22,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:22,196 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:22,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:22,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,196 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,196 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,196 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,196 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:22,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:22,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:22,196 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:22,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:22,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838982254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:22,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838982255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:22,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838982256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,296 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:22,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:22,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:22,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:22,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:22,299 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:22,299 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:22,299 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:22,299 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:22,300 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,300 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,301 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:22,301 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:22,301 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,301 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:22,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:22,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,302 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:22,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:22,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:22,302 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:22,302 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:22,302 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:22,303 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:22,303 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:22,303 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:22,303 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:22,303 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:22,303 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:22,303 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:22,303 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:22,303 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,303 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:22,303 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:22,306 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,306 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,306 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,306 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:22,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@411ca271[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:22,308 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:22,308 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,309 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,310 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:22,310 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:22,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:22,310 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:22,311 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,311 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,311 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:22,311 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:22,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:22,313 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:22,313 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:22,313 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:22,313 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:22,313 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:22,315 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,315 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,315 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,315 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,316 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,319 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,320 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,320 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,320 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:22,320 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:22,320 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:22,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:22,320 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,320 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,320 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:22,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:22,320 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:22,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:22,320 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,320 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:22,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:22,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:22,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:22,368 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=259, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/d25dc7673a064187b494ea181160b257 2023-03-26 13:55:22,420 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:22,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:22,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:22,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:22,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:22,423 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:22,423 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:22,423 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:22,423 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:22,424 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,424 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,425 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:22,425 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:22,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:22,425 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,425 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:22,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:22,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:22,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:22,426 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:22,426 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:22,426 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:22,426 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:22,426 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:22,426 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:22,427 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:22,427 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:22,427 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:22,427 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:22,427 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:22,427 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,427 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:22,427 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:22,428 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,429 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,429 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:22,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79e7679a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:22,430 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:22,430 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,437 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:22,437 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:22,437 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:22,437 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,437 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:22,438 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:22,438 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:22,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,438 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:22,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:22,441 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:22,441 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:22,441 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:22,441 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:22,441 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:22,443 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,443 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,443 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:22,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:22,443 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:22,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,444 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,446 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,446 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:22,446 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:22,446 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:22,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:22,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:22,446 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:22,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:22,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:22,446 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,446 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:22,447 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:22,447 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,447 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,447 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,447 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:22,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:22,547 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:22,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:22,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:22,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:22,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:22,550 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:22,550 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:22,550 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:22,550 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:22,550 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,550 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,552 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:22,552 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:22,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:22,553 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,553 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:22,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:22,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:22,554 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:22,554 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:22,554 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:22,554 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:22,554 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:22,554 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:22,554 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:22,554 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:22,554 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:22,554 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:22,554 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:22,554 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:22,554 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:22,554 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,554 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:22,554 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:22,556 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,556 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,556 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,556 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:22,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e4fbad[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:22,557 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:22,557 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,558 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,558 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:22,558 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:22,559 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:22,558 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,559 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:22,560 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:22,560 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,560 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,560 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:22,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:22,562 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:22,562 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:22,562 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:22,562 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:22,562 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:22,565 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,565 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,565 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:22,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:22,565 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,567 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,569 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,569 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,569 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:22,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:22,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:22,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:22,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:22,569 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:22,569 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:22,569 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:22,569 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:22,569 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,569 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,569 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,569 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,569 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:22,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:22,669 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:22,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:22,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:22,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:22,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:22,672 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:22,672 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:22,672 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:22,672 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:22,673 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,673 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,674 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,674 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:22,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:22,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:22,674 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,674 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:22,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:22,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:22,676 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:22,676 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:22,676 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:22,676 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:22,676 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:22,676 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:22,676 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:22,676 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:22,676 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:22,676 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:22,676 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:22,676 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,676 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:22,676 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:22,677 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,678 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,678 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,678 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:22,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7977c526[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:22,679 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:22,679 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,681 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,681 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:22,681 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:22,681 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,681 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:22,681 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:22,682 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:22,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:22,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:22,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:22,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:22,684 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:22,685 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:22,685 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:22,685 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:22,685 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:22,686 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,686 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,686 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:22,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,689 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:22,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:22,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:22,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:22,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:22,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:22,691 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:22,691 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:22,691 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:22,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:22,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:22,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:22,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838982760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:22,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838982766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:22,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838982769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,779 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/70192155db694a76b032e983888e5429 2023-03-26 13:55:22,791 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:22,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:22,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:22,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:22,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:22,794 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:22,795 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:22,795 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:22,795 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:22,795 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,795 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,801 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:22,801 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:22,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:22,802 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,802 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:22,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:22,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:22,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:22,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:22,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:22,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:22,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:22,803 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:22,803 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:22,803 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:22,803 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:22,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:22,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:22,804 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,804 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:22,804 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:22,805 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,805 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,805 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,805 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:22,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ddcff53[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:22,807 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:22,807 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,808 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,808 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:22,808 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:22,808 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:22,810 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,810 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:22,810 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,810 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:22,811 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,811 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:22,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:22,813 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:22,813 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:22,813 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:22,813 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:22,813 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:22,818 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,818 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,818 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:22,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,818 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,820 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:22,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:22,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:22,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:22,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:22,822 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:22,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,822 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:22,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:22,822 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:22,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:22,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,823 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,923 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:22,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:22,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:22,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:22,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:22,926 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:22,926 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:22,926 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:22,926 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:22,926 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,926 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,928 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:22,928 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:22,928 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,928 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:22,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:22,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:22,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:22,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:22,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:22,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:22,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:22,930 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:22,930 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:22,930 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:22,930 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:22,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:22,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:22,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:22,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:22,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,931 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,932 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:22,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f4598c5[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:22,933 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:22,933 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,935 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,935 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:22,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:22,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:22,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:22,936 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,936 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:22,936 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:22,937 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:22,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:22,939 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:22,939 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:22,939 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:22,939 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:22,939 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:22,940 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,941 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:22,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,941 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:22,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:22,941 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:22,942 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:22,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:22,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:22,945 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:22,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:22,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:22,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:22,946 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:22,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:22,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:22,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:22,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:22,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:22,946 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:22,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:22,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:22,946 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:23,046 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:23,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:23,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:23,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:23,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:23,049 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:23,049 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:23,049 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:23,049 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:23,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,052 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,052 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:23,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:23,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:23,053 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,053 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:23,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:23,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:23,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:23,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:23,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:23,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:23,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:23,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:23,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:23,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:23,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:23,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:23,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:23,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:23,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:23,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:23,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:23,056 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,056 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:23,056 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,056 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:23,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:23,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:23,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fdc3113[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:23,058 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:23,058 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,059 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,059 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:23,059 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:23,059 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:23,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:23,060 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:23,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:23,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:23,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:23,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:23,062 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:23,062 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:23,062 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:23,062 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:23,062 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:23,065 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,065 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,065 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:23,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:23,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:23,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:23,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:23,065 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:23,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,068 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,068 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,068 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:23,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:23,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:23,068 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:23,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:23,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:23,068 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:23,069 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:23,069 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:23,068 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,069 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,069 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:23,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:23,069 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,069 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,169 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:23,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:23,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:23,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:23,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:23,172 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:23,172 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:23,172 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:23,172 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:23,172 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,172 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,173 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,173 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:23,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:23,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:23,174 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,174 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:23,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:23,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:23,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:23,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:23,175 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:23,175 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:23,175 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:23,175 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:23,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:23,176 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,177 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:23,177 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,177 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:23,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:23,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78c58d3b[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:23,178 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:23,178 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,181 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,181 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:23,181 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:23,181 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:23,181 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,181 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:23,182 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,182 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:23,182 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:23,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:23,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:23,183 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:23,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:23,189 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/7c75ddea9c64411aa23fe0d596f840f8 2023-03-26 13:55:23,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:23,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:23,190 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:23,191 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:23,191 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:23,191 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:23,191 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:23,193 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,193 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:23,193 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:23,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:23,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,194 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/d25dc7673a064187b494ea181160b257 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/d25dc7673a064187b494ea181160b257 2023-03-26 13:55:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:23,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:23,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,195 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,196 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,199 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/d25dc7673a064187b494ea181160b257, entries=250, sequenceid=259, filesize=39.0 K 2023-03-26 13:55:23,200 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/70192155db694a76b032e983888e5429 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/70192155db694a76b032e983888e5429 2023-03-26 13:55:23,203 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,203 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,203 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:23,203 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,203 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:23,203 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:23,203 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:23,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:23,203 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:23,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:23,204 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,204 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,204 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:23,204 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:23,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:23,206 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/70192155db694a76b032e983888e5429, entries=150, sequenceid=259, filesize=11.9 K 2023-03-26 13:55:23,217 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/7c75ddea9c64411aa23fe0d596f840f8 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/7c75ddea9c64411aa23fe0d596f840f8 2023-03-26 13:55:23,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/7c75ddea9c64411aa23fe0d596f840f8, entries=150, sequenceid=259, filesize=11.9 K 2023-03-26 13:55:23,220 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for a9871eaedebb91319d17cf11612be1e0 in 1673ms, sequenceid=259, compaction requested=true 2023-03-26 13:55:23,220 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:23,220 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:23,220 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-03-26 13:55:23,220 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-03-26 13:55:23,220 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:23,221 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:23,223 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 273100 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-03-26 13:55:23,223 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 120516 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-03-26 13:55:23,223 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a9871eaedebb91319d17cf11612be1e0/A is initiating minor compaction (all files) 2023-03-26 13:55:23,223 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a9871eaedebb91319d17cf11612be1e0/B is initiating minor compaction (all files) 2023-03-26 13:55:23,223 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a9871eaedebb91319d17cf11612be1e0/A in TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:23,223 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9871eaedebb91319d17cf11612be1e0/B in TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:23,223 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/f21b3ba02e0c4d12a7b38cc98ff3b7d8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/01b7d924b1fb4de2b340be18c61d73f5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/0437d49d2c2e4bc09c378c1c0ceb9646, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/12fdc39eff514e01bf482ee463d5b692, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5989223eab014cbdb1a33966a307f891, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/3b10b87c8599410a846f708a8fb58650, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/29c97c21827f4d2ea89a4a3d30c33460, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/e3a2b1e18471428c8f280078d383e45d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/b0ca98eb4c7440f29cc1ce72ec7ca083, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/d25dc7673a064187b494ea181160b257] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp, totalSize=266.7 K 2023-03-26 13:55:23,223 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/cdbdd69f04e14654a50892575667f086, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/ec196239ef104244aacfe16f0945cfd4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/cbe17c6cf273476cb8a4252beae59b99, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/515e6f451db846d8bb73ceb384c7b202, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/f3e527038832454dab9be1b9d4b0a85c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/c0c3d832cc124f5689c21c97010ae868, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/82d03ddcd7dc45de8e5b044818ad6f44, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/c09d0d50f46941cd9f12f3779ef15e2e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/0ca0c294676c49969508c2eae85999ae, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/70192155db694a76b032e983888e5429] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp, totalSize=117.7 K 2023-03-26 13:55:23,224 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting f21b3ba02e0c4d12a7b38cc98ff3b7d8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1679838912344 2023-03-26 13:55:23,224 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting cdbdd69f04e14654a50892575667f086, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1679838912344 2023-03-26 13:55:23,224 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 01b7d924b1fb4de2b340be18c61d73f5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1679838913523 2023-03-26 13:55:23,224 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting ec196239ef104244aacfe16f0945cfd4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1679838913523 2023-03-26 13:55:23,224 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 0437d49d2c2e4bc09c378c1c0ceb9646, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1679838914662 2023-03-26 13:55:23,224 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting cbe17c6cf273476cb8a4252beae59b99, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1679838914662 2023-03-26 13:55:23,224 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 12fdc39eff514e01bf482ee463d5b692, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1679838915826 2023-03-26 13:55:23,225 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 515e6f451db846d8bb73ceb384c7b202, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1679838915826 2023-03-26 13:55:23,225 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 5989223eab014cbdb1a33966a307f891, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1679838916962 2023-03-26 13:55:23,225 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting f3e527038832454dab9be1b9d4b0a85c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1679838916962 2023-03-26 13:55:23,225 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 3b10b87c8599410a846f708a8fb58650, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1679838918146 2023-03-26 13:55:23,225 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting c0c3d832cc124f5689c21c97010ae868, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1679838918146 2023-03-26 13:55:23,225 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 29c97c21827f4d2ea89a4a3d30c33460, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1679838918483 2023-03-26 13:55:23,225 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 82d03ddcd7dc45de8e5b044818ad6f44, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1679838918483 2023-03-26 13:55:23,226 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting e3a2b1e18471428c8f280078d383e45d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1679838919691 2023-03-26 13:55:23,226 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting c09d0d50f46941cd9f12f3779ef15e2e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1679838919691 2023-03-26 13:55:23,226 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting b0ca98eb4c7440f29cc1ce72ec7ca083, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679838920853 2023-03-26 13:55:23,226 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 0ca0c294676c49969508c2eae85999ae, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679838920853 2023-03-26 13:55:23,226 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting d25dc7673a064187b494ea181160b257, keycount=250, bloomtype=ROW, size=39.0 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679838921514 2023-03-26 13:55:23,227 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 70192155db694a76b032e983888e5429, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679838921528 2023-03-26 13:55:23,246 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a9871eaedebb91319d17cf11612be1e0#B#compaction#399 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:23,265 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a9871eaedebb91319d17cf11612be1e0#A#compaction#400 average throughput is 1.16 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:23,304 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:23,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:23,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:23,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:23,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:23,309 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:23,309 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:23,309 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:23,309 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:23,309 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,309 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,311 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,311 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:23,312 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:23,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:23,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:23,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:23,316 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:23,316 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:23,316 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:23,316 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:23,316 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:23,316 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:23,316 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:23,317 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-26 13:55:23,317 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:23,317 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:23,317 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:23,317 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:23,317 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:23,317 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:23,337 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032626bd286001554ec293475e28fc0742d0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032626bd286001554ec293475e28fc0742d0 2023-03-26 13:55:23,669 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/48b33fd5bca5466d949c8a3b7d112131 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/48b33fd5bca5466d949c8a3b7d112131 2023-03-26 13:55:23,673 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in a9871eaedebb91319d17cf11612be1e0/B of a9871eaedebb91319d17cf11612be1e0 into 48b33fd5bca5466d949c8a3b7d112131(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:23,673 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:23,673 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., storeName=a9871eaedebb91319d17cf11612be1e0/B, priority=6, startTime=1679838923220; duration=0sec 2023-03-26 13:55:23,673 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:23,673 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-03-26 13:55:23,675 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 120516 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-03-26 13:55:23,675 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a9871eaedebb91319d17cf11612be1e0/C is initiating minor compaction (all files) 2023-03-26 13:55:23,675 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9871eaedebb91319d17cf11612be1e0/C in TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:23,675 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/f378b8b6ea604720b75114399539156a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/1e823dee8e8b460db53be47f09b4e16c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/1853a75ba65d42de8604d30f0cd5b31b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/54248848bbac4caa8f61092bc8b4ecf7, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/28959ac0406f47a5bc2e7af6274c3433, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/3e3ff27aff484d0494be6293e74c52ca, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/3b2876f4916f45f7a1c5861894903913, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/dc2d9621b6684c85b6f885b31b2ad9aa, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/be4253f0533a4d3fb8fbc154e9100cb4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/7c75ddea9c64411aa23fe0d596f840f8] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp, totalSize=117.7 K 2023-03-26 13:55:23,675 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting f378b8b6ea604720b75114399539156a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=112, earliestPutTs=1679838912344 2023-03-26 13:55:23,676 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 1e823dee8e8b460db53be47f09b4e16c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1679838913523 2023-03-26 13:55:23,676 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 1853a75ba65d42de8604d30f0cd5b31b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1679838914662 2023-03-26 13:55:23,676 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 54248848bbac4caa8f61092bc8b4ecf7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1679838915826 2023-03-26 13:55:23,676 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 28959ac0406f47a5bc2e7af6274c3433, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=180, earliestPutTs=1679838916962 2023-03-26 13:55:23,677 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 3e3ff27aff484d0494be6293e74c52ca, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1679838918146 2023-03-26 13:55:23,677 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 3b2876f4916f45f7a1c5861894903913, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1679838918483 2023-03-26 13:55:23,677 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting dc2d9621b6684c85b6f885b31b2ad9aa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1679838919691 2023-03-26 13:55:23,678 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting be4253f0533a4d3fb8fbc154e9100cb4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679838920853 2023-03-26 13:55:23,678 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 7c75ddea9c64411aa23fe0d596f840f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679838921528 2023-03-26 13:55:23,690 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/e76c673de3f2468a956837ff2c57acdc as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/e76c673de3f2468a956837ff2c57acdc 2023-03-26 13:55:23,694 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in a9871eaedebb91319d17cf11612be1e0/A of a9871eaedebb91319d17cf11612be1e0 into e76c673de3f2468a956837ff2c57acdc(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:23,694 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:23,694 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., storeName=a9871eaedebb91319d17cf11612be1e0/A, priority=6, startTime=1679838923220; duration=0sec 2023-03-26 13:55:23,694 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:23,695 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a9871eaedebb91319d17cf11612be1e0#C#compaction#402 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:23,746 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=277, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/df750aec4ac849ef91e1dd7f424848f5 2023-03-26 13:55:23,760 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/541f0ee4985b4c9cbbfa5ac1a434782b 2023-03-26 13:55:23,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:23,775 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:23,797 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/143dc90fc6c6436d9f4042ef83fc185a 2023-03-26 13:55:23,800 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/df750aec4ac849ef91e1dd7f424848f5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/df750aec4ac849ef91e1dd7f424848f5 2023-03-26 13:55:23,803 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/df750aec4ac849ef91e1dd7f424848f5, entries=150, sequenceid=277, filesize=25.4 K 2023-03-26 13:55:23,804 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/541f0ee4985b4c9cbbfa5ac1a434782b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/541f0ee4985b4c9cbbfa5ac1a434782b 2023-03-26 13:55:23,807 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/541f0ee4985b4c9cbbfa5ac1a434782b, entries=150, sequenceid=277, filesize=11.9 K 2023-03-26 13:55:23,807 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/143dc90fc6c6436d9f4042ef83fc185a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/143dc90fc6c6436d9f4042ef83fc185a 2023-03-26 13:55:23,811 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/143dc90fc6c6436d9f4042ef83fc185a, entries=150, sequenceid=277, filesize=11.9 K 2023-03-26 13:55:23,811 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=53.67 KB/54960 for a9871eaedebb91319d17cf11612be1e0 in 494ms, sequenceid=277, compaction requested=false 2023-03-26 13:55:23,811 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:23,812 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:23,812 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:23,813 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,813 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:23,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:23,814 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,814 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:23,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:23,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:23,814 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 13:55:23,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:23,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:23,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:23,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:23,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:23,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:23,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:23,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:23,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:23,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4259d317[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:23,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:23,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,817 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,817 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:23,817 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:23,817 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:23,817 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,817 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:23,818 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:23,818 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:23,818 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:23,818 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:23,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:23,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:23,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:23,821 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:23,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:23,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:23,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:23,821 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:23,823 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,823 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:23,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,823 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:23,824 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326382594dc29c241d6aad8a6fdda17b2ba to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326382594dc29c241d6aad8a6fdda17b2ba 2023-03-26 13:55:23,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:23,824 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:23,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:23,825 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,828 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:23,828 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:23,828 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:23,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:23,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:23,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:23,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:23,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:23,831 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,831 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,831 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,831 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,831 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,831 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,831 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:23,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:23,831 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:23,831 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:23,832 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=291, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/f8cda67799a24e668a2efec2d39f088f 2023-03-26 13:55:23,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:23,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838983857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,871 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:23,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838983858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,871 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:23,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838983859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,931 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:23,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:23,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:23,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:23,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:23,934 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:23,934 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:23,934 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:23,934 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:23,935 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,935 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,936 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:23,936 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:23,936 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,936 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:23,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:23,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:23,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:23,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:23,937 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:23,937 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:23,937 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:23,938 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:23,938 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:23,938 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:23,938 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:23,938 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:23,938 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:23,938 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:23,938 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:23,938 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,938 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:23,938 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:23,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,939 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:23,940 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,940 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:23,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:23,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71e9e829[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:23,941 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:23,941 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,943 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:23,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:23,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:23,943 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,943 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:23,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,944 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:23,945 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:23,945 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:23,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:23,947 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:23,947 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:23,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:23,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:23,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:23,947 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:23,948 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,948 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:23,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:23,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,948 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:23,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:23,949 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:23,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:23,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:23,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,950 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:23,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:23,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,953 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,953 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:23,953 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,953 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:23,953 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:23,953 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:23,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:23,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:23,953 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:23,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:23,953 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,953 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:23,953 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:23,953 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:23,953 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:23,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:23,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:23,973 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:23,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838983969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:23,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838983972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:23,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:23,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838983972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,053 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:24,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:24,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:24,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:24,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:24,057 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:24,057 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:24,057 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:24,057 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:24,057 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,057 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:24,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,059 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:24,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:24,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:24,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:24,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:24,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:24,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:24,060 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:24,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:24,061 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:24,061 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:24,061 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:24,061 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:24,061 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:24,061 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:24,061 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:24,061 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:24,061 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:24,061 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:24,061 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:24,061 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:24,061 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,061 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:24,061 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:24,063 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,063 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:24,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:24,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:24,063 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:24,063 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:24,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:24,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:24,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:24,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:24,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:24,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fe31bae[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:24,064 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:24,065 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:24,066 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:24,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:24,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:24,066 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:24,066 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:24,066 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:24,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:24,067 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:24,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:24,067 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:24,067 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:24,067 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:24,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:24,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:24,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:24,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:24,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:24,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:24,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:24,071 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:24,071 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:24,071 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:24,071 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:24,071 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:24,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:24,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:24,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:24,074 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:24,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:24,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:24,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,075 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,075 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:24,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,075 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:24,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:24,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:24,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:24,085 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:24,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:24,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:24,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:24,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:24,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:24,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:24,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:24,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:24,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:24,086 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:24,086 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:24,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:24,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,102 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/34fa8e8e75f64c38907c9499f565fa20 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/34fa8e8e75f64c38907c9499f565fa20 2023-03-26 13:55:24,106 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in a9871eaedebb91319d17cf11612be1e0/C of a9871eaedebb91319d17cf11612be1e0 into 34fa8e8e75f64c38907c9499f565fa20(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:24,106 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:24,106 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., storeName=a9871eaedebb91319d17cf11612be1e0/C, priority=6, startTime=1679838923221; duration=0sec 2023-03-26 13:55:24,106 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:24,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:24,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838984175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:24,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838984177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:24,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838984177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,186 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:24,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:24,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:24,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:24,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:24,189 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:24,189 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:24,189 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:24,189 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:24,190 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,190 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:24,191 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:24,191 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:24,191 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,191 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:24,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:24,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:24,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:24,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:24,192 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:24,192 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:24,193 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:24,193 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:24,193 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:24,193 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:24,193 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:24,193 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:24,193 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:24,193 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:24,193 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:24,193 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,193 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:24,193 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:24,197 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:24,197 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:24,197 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:24,197 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:24,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:24,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b9be73c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:24,199 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:24,199 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:24,200 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:24,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:24,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:24,200 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:24,200 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:24,200 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:24,200 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,200 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:24,201 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:24,201 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:24,201 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,201 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:24,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:24,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:24,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:24,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:24,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:24,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:24,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:24,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:24,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:24,204 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:24,204 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:24,204 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:24,204 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:24,204 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:24,205 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,205 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:24,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:24,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,205 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:24,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:24,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:24,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:24,206 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:24,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:24,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:24,207 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:24,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,211 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,211 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:24,211 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:24,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:24,211 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:24,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:24,211 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:24,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:24,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:24,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:24,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:24,211 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:24,211 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,211 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:24,211 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:24,211 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:24,211 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:24,211 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,239 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/0cae603fad67442c8bc0cfba7228f593 2023-03-26 13:55:24,247 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=291 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/9b63d39fadf8403eab6212ec8079cd8f 2023-03-26 13:55:24,250 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/f8cda67799a24e668a2efec2d39f088f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/f8cda67799a24e668a2efec2d39f088f 2023-03-26 13:55:24,253 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/f8cda67799a24e668a2efec2d39f088f, entries=250, sequenceid=291, filesize=39.2 K 2023-03-26 13:55:24,254 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/0cae603fad67442c8bc0cfba7228f593 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/0cae603fad67442c8bc0cfba7228f593 2023-03-26 13:55:24,257 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/0cae603fad67442c8bc0cfba7228f593, entries=150, sequenceid=291, filesize=11.9 K 2023-03-26 13:55:24,257 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/9b63d39fadf8403eab6212ec8079cd8f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/9b63d39fadf8403eab6212ec8079cd8f 2023-03-26 13:55:24,260 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/9b63d39fadf8403eab6212ec8079cd8f, entries=150, sequenceid=291, filesize=11.9 K 2023-03-26 13:55:24,261 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for a9871eaedebb91319d17cf11612be1e0 in 447ms, sequenceid=291, compaction requested=true 2023-03-26 13:55:24,261 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:24,261 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:24,261 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:24,261 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:24,261 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:24,261 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:24,262 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92718 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:24,262 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:24,262 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a9871eaedebb91319d17cf11612be1e0/A is initiating minor compaction (all files) 2023-03-26 13:55:24,262 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a9871eaedebb91319d17cf11612be1e0/B is initiating minor compaction (all files) 2023-03-26 13:55:24,262 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a9871eaedebb91319d17cf11612be1e0/A in TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:24,262 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9871eaedebb91319d17cf11612be1e0/B in TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:24,262 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/e76c673de3f2468a956837ff2c57acdc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/df750aec4ac849ef91e1dd7f424848f5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/f8cda67799a24e668a2efec2d39f088f] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp, totalSize=90.5 K 2023-03-26 13:55:24,262 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/48b33fd5bca5466d949c8a3b7d112131, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/541f0ee4985b4c9cbbfa5ac1a434782b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/0cae603fad67442c8bc0cfba7228f593] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp, totalSize=36.2 K 2023-03-26 13:55:24,262 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting e76c673de3f2468a956837ff2c57acdc, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679838921528 2023-03-26 13:55:24,262 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 48b33fd5bca5466d949c8a3b7d112131, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679838921528 2023-03-26 13:55:24,263 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting df750aec4ac849ef91e1dd7f424848f5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1679838921599 2023-03-26 13:55:24,263 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 541f0ee4985b4c9cbbfa5ac1a434782b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1679838921599 2023-03-26 13:55:24,263 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting f8cda67799a24e668a2efec2d39f088f, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1679838923773 2023-03-26 13:55:24,263 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 0cae603fad67442c8bc0cfba7228f593, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1679838923807 2023-03-26 13:55:24,268 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a9871eaedebb91319d17cf11612be1e0#B#compaction#409 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:24,271 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a9871eaedebb91319d17cf11612be1e0#A#compaction#408 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:24,286 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/2d0b2b733a10439aa353132584657a48 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/2d0b2b733a10439aa353132584657a48 2023-03-26 13:55:24,289 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9871eaedebb91319d17cf11612be1e0/A of a9871eaedebb91319d17cf11612be1e0 into 2d0b2b733a10439aa353132584657a48(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:24,289 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:24,289 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., storeName=a9871eaedebb91319d17cf11612be1e0/A, priority=13, startTime=1679838924261; duration=0sec 2023-03-26 13:55:24,289 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:24,289 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:24,290 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37054 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:24,290 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a9871eaedebb91319d17cf11612be1e0/C is initiating minor compaction (all files) 2023-03-26 13:55:24,290 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a9871eaedebb91319d17cf11612be1e0/C in TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:24,290 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/34fa8e8e75f64c38907c9499f565fa20, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/143dc90fc6c6436d9f4042ef83fc185a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/9b63d39fadf8403eab6212ec8079cd8f] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp, totalSize=36.2 K 2023-03-26 13:55:24,290 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 34fa8e8e75f64c38907c9499f565fa20, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=259, earliestPutTs=1679838921528 2023-03-26 13:55:24,291 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 143dc90fc6c6436d9f4042ef83fc185a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1679838921599 2023-03-26 13:55:24,291 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 9b63d39fadf8403eab6212ec8079cd8f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1679838923807 2023-03-26 13:55:24,295 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a9871eaedebb91319d17cf11612be1e0#C#compaction#410 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:24,311 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:24,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:24,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:24,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:24,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:24,314 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:24,314 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:24,314 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:24,314 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:24,315 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,315 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:24,316 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,316 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:24,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:24,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:24,316 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,316 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:24,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:24,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:24,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:24,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:24,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:24,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:24,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:24,317 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:24,317 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:24,317 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:24,317 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:24,317 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:24,317 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:24,317 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:24,317 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-26 13:55:24,318 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:24,318 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:24,318 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:24,318 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:24,318 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:24,318 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:24,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:24,484 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:24,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:24,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838984497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:24,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838984498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:24,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838984498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,607 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:24,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838984604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,607 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:24,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838984604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:24,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838984607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,681 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/2a3f9cc227334101b4685e3f99948904 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/2a3f9cc227334101b4685e3f99948904 2023-03-26 13:55:24,687 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9871eaedebb91319d17cf11612be1e0/B of a9871eaedebb91319d17cf11612be1e0 into 2a3f9cc227334101b4685e3f99948904(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:24,687 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:24,687 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., storeName=a9871eaedebb91319d17cf11612be1e0/B, priority=13, startTime=1679838924261; duration=0sec 2023-03-26 13:55:24,687 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:24,717 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/738a41875770413b8d26b23a91a7f5e5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/738a41875770413b8d26b23a91a7f5e5 2023-03-26 13:55:24,724 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9871eaedebb91319d17cf11612be1e0/C of a9871eaedebb91319d17cf11612be1e0 into 738a41875770413b8d26b23a91a7f5e5(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:24,724 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:24,724 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., storeName=a9871eaedebb91319d17cf11612be1e0/C, priority=13, startTime=1679838924261; duration=0sec 2023-03-26 13:55:24,724 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:24,743 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032620800782c94a4faa82fd214c216e1b31 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032620800782c94a4faa82fd214c216e1b31 2023-03-26 13:55:24,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:24,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838984809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:24,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838984809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:24,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:24,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838984810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:25,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:25,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:25,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838985115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:25,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838985115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:25,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:25,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838985117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:25,182 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=313, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/1f1351c60489466ab298f68ef352d701 2023-03-26 13:55:25,592 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/d3bda2bbb3ff4d79b9f859f34b659956 2023-03-26 13:55:25,626 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:25,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838985622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:25,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:25,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838985623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:25,628 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:25,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838985624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,015 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/eda66233b912405a8e461214b41f0e27 2023-03-26 13:55:26,019 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/1f1351c60489466ab298f68ef352d701 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/1f1351c60489466ab298f68ef352d701 2023-03-26 13:55:26,022 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/1f1351c60489466ab298f68ef352d701, entries=150, sequenceid=313, filesize=25.4 K 2023-03-26 13:55:26,023 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/d3bda2bbb3ff4d79b9f859f34b659956 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/d3bda2bbb3ff4d79b9f859f34b659956 2023-03-26 13:55:26,026 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/d3bda2bbb3ff4d79b9f859f34b659956, entries=150, sequenceid=313, filesize=11.9 K 2023-03-26 13:55:26,027 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/eda66233b912405a8e461214b41f0e27 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/eda66233b912405a8e461214b41f0e27 2023-03-26 13:55:26,030 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/eda66233b912405a8e461214b41f0e27, entries=150, sequenceid=313, filesize=11.9 K 2023-03-26 13:55:26,031 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a9871eaedebb91319d17cf11612be1e0 in 1714ms, sequenceid=313, compaction requested=false 2023-03-26 13:55:26,031 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:26,031 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:26,031 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:26,031 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:26,031 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,031 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:26,031 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:26,032 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,032 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,033 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,033 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:26,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77d65233[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:26,034 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:26,034 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,035 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,035 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:26,035 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:26,035 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:26,035 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,035 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:26,038 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:26,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:26,040 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:26,040 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:26,040 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:26,040 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:26,040 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:26,040 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:26,042 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,042 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,042 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:26,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,042 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:26,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:26,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,044 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:26,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:26,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:26,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,046 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:26,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:26,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:26,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,046 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:26,047 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,047 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:26,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:26,147 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:26,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:26,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:26,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:26,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:26,150 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:26,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:26,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:26,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:26,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,151 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:26,151 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:26,152 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,152 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:26,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:26,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:26,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:26,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:26,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:26,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:26,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:26,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:26,153 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:26,153 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:26,153 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 13:55:26,153 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:26,153 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:26,153 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:26,153 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:26,153 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:26,153 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:26,164 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032655837411e86e4439810075b0e69f2779 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032655837411e86e4439810075b0e69f2779 2023-03-26 13:55:26,168 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=327, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/1180852c24f84c9e839990eb75eb1b74 2023-03-26 13:55:26,178 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/e050f3e825ea468ca458f82d28432e40 2023-03-26 13:55:26,195 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/6fad7742ceef40c8949422571d96cb61 2023-03-26 13:55:26,198 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/1180852c24f84c9e839990eb75eb1b74 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/1180852c24f84c9e839990eb75eb1b74 2023-03-26 13:55:26,209 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/1180852c24f84c9e839990eb75eb1b74, entries=150, sequenceid=327, filesize=25.4 K 2023-03-26 13:55:26,210 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/e050f3e825ea468ca458f82d28432e40 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/e050f3e825ea468ca458f82d28432e40 2023-03-26 13:55:26,214 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/e050f3e825ea468ca458f82d28432e40, entries=150, sequenceid=327, filesize=11.9 K 2023-03-26 13:55:26,215 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/6fad7742ceef40c8949422571d96cb61 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/6fad7742ceef40c8949422571d96cb61 2023-03-26 13:55:26,231 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/6fad7742ceef40c8949422571d96cb61, entries=150, sequenceid=327, filesize=11.9 K 2023-03-26 13:55:26,232 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for a9871eaedebb91319d17cf11612be1e0 in 79ms, sequenceid=327, compaction requested=true 2023-03-26 13:55:26,232 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:26,232 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:26,232 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:26,232 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:26,232 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,232 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:26,232 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:26,235 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,235 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,235 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,235 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:26,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49405440[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:26,237 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:26,237 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,238 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,239 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:26,239 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:26,239 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:26,239 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,239 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:26,241 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,241 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:26,241 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:26,241 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:26,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:26,244 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:26,244 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:26,244 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:26,244 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:26,244 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:26,245 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,246 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,246 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,246 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,247 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,251 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:26,251 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:26,251 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:26,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:26,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:26,251 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:26,251 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:26,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:26,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:26,251 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,251 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:26,251 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,251 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:26,252 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,252 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,252 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,352 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:26,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:26,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:26,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:26,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:26,355 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:26,355 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:26,355 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:26,355 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:26,356 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,356 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,357 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,357 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:26,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:26,357 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,357 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:26,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:26,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:26,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:26,359 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:26,359 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:26,359 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:26,359 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:26,364 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,364 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,364 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,364 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:26,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e4908ad[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:26,366 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:26,366 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,367 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,367 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:26,367 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:26,367 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:26,367 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,367 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:26,368 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,368 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:26,368 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,368 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:26,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:26,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:26,371 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:26,371 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:26,371 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:26,371 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:26,371 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:26,373 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,373 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,373 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:26,374 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,375 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,377 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,377 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:26,377 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:26,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:26,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:26,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:26,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:26,377 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:26,377 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,377 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:26,377 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:26,377 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:26,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:26,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:26,377 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,378 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,378 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,378 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,478 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:26,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:26,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:26,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:26,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:26,483 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:26,483 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:26,484 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:26,484 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:26,484 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,484 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,485 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,485 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:26,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:26,485 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,485 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:26,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:26,486 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:26,486 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:26,486 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:26,487 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:26,487 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:26,487 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:26,487 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:26,487 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:26,487 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:26,487 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:26,487 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:26,488 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:26,488 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:26,488 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,488 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:26,488 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:26,489 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,489 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,490 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,490 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:26,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5acfa25f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:26,491 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:26,492 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,494 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:26,495 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,495 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:26,495 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:26,496 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:26,496 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,496 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,496 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:26,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:26,499 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:26,499 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:26,499 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:26,499 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:26,499 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:26,501 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,501 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,501 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:26,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:26,501 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,503 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:26,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,505 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:26,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:26,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:26,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:26,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:26,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:26,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:26,506 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:26,506 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:26,606 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:26,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:26,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:26,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:26,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:26,610 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:26,610 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:26,610 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:26,610 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:26,610 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,610 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:26,612 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:26,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:26,613 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,613 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:26,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:26,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,614 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:26,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:26,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:26,620 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:26,620 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:26,620 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:26,620 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:26,620 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:26,620 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:26,621 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:26,621 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:26,621 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:26,621 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:26,621 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:26,621 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,621 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:26,621 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:26,622 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,622 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,623 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,623 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:26,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:26,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f4d24ff[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:26,626 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:26,626 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,627 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,627 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:26,627 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:26,627 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:26,627 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,627 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:26,629 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:26,629 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:26,629 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,629 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:26,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:26,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:26,632 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:26,632 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:26,632 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:26,632 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:26,632 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:26,636 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,636 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:26,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,636 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,638 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,639 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,641 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:26,641 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,641 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:26,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:26,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:26,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:26,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:26,641 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:26,641 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,642 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,642 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,642 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,642 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,642 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:26,642 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:26,642 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:26,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:26,742 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:26,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:26,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:26,743 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:55:26,743 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:26,743 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:26,743 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:26,743 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:26,744 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:26,744 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:26,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:26,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:26,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:26,746 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:26,746 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:26,746 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:26,746 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:26,746 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,746 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,747 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:26,747 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:26,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:26,748 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,748 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:26,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:26,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,749 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:26,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:26,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:26,749 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:26,749 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:26,749 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:26,749 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:26,749 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:26,750 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:26,750 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:26,750 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:26,750 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:26,750 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:26,755 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,755 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,755 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,755 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:26,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:26,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54e48102[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:26,757 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:26,757 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,766 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,766 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:26,766 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:26,766 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:26,766 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,767 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:26,768 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:26,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:26,771 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:26,771 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:26,771 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:26,771 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:26,771 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:26,772 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,772 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,772 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:26,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,773 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,774 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:26,780 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:26,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:26,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:26,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:26,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:26,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:26,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:26,781 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:26,781 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:26,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:26,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:26,781 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,782 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1679838926765, max timestamp=1679838926784Num uniques 859; 2023-03-26 13:55:26,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-26 13:55:26,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1679838926765, max timestamp=1679838926784Num uniques 859; 2023-03-26 13:55:26,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-26 13:55:26,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838926766, max timestamp=1679838926784Num uniques 859; 2023-03-26 13:55:26,787 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:55:26,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:26,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838986817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:26,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838986818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:26,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838986820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,881 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:26,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:26,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:26,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:26,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:26,884 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:26,884 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:26,884 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:26,884 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:26,885 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,885 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,886 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:26,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:26,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:26,886 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,886 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,886 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:26,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:26,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:26,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:26,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:26,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:26,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:26,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:26,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:26,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:26,888 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:26,888 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:26,888 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:26,888 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:26,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:26,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:26,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:26,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:26,890 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,890 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:26,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62def753[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:26,891 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:26,891 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,892 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,893 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:26,893 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:26,893 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:26,893 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,893 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:26,895 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:26,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:26,895 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:26,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:26,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:26,899 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:26,899 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:26,899 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:26,899 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:26,899 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:26,900 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,900 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,900 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:26,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:26,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:26,901 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,903 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,905 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:26,905 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,905 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:26,905 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:26,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:26,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:26,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:26,905 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:26,905 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:26,905 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:26,905 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:26,905 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,905 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,905 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:26,905 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:26,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:26,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:26,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:26,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838986925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:26,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838986931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:26,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:26,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838986931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,005 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:27,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:27,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:27,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:27,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:27,008 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:27,008 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:27,009 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:27,009 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:27,009 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,009 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,010 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,010 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:27,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:27,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:27,010 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,010 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:27,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,011 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:27,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:27,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:27,012 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:27,012 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:27,012 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:27,012 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:27,012 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:27,015 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,015 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:27,015 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,015 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:27,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:27,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:27,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45246c0d[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:27,018 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:27,018 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,019 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:27,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:27,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:27,020 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,020 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:27,021 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:27,021 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:27,021 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:27,021 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:27,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:27,023 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:27,024 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:27,024 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:27,024 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:27,024 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:27,025 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,025 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,025 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:27,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:27,026 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,026 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,030 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:27,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:27,030 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:27,030 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:27,030 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,030 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:27,030 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:27,030 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:27,030 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,030 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,030 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,030 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,030 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:27,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:27,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:27,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:27,130 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:27,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:27,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:27,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:27,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:27,134 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:27,134 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:27,134 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:27,134 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:27,134 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,134 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:27,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838987129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:27,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:27,135 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,136 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,136 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:27,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:27,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:27,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:27,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:27,137 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:27,137 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:27,138 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:27,138 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:27,138 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:27,138 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:27,138 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,138 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:27,138 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:27,139 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,139 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:27,139 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,139 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:27,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:27,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:27,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838987136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:27,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:27,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33b8c28a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:27,141 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:27,141 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,142 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:27,142 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:27,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:27,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838987136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,147 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:27,147 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,147 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:27,147 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:27,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:27,150 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:27,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:27,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:27,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:27,150 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:27,151 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,151 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,151 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:27,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:27,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:27,152 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:27,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,153 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,155 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:27,155 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:27,155 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:27,155 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,155 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,155 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,155 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,155 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,155 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:27,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:27,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:27,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:27,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:27,157 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:27,157 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:27,163 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326b5b949be63bd4990bbb3c4ff7f14afff to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326b5b949be63bd4990bbb3c4ff7f14afff 2023-03-26 13:55:27,257 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:27,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:27,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:27,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:27,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:27,260 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:27,260 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:27,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:27,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:27,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,264 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:27,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:27,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:27,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:27,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:27,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:27,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:27,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:27,265 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:27,265 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:27,266 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:27,266 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:27,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:27,267 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:27,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:27,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43a4b96e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:27,274 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:27,275 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,276 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,276 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:27,276 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:27,276 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:27,276 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:27,278 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,278 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:27,278 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:27,278 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:27,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:27,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:27,280 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:27,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:27,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:27,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:27,280 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:27,283 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,283 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,283 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:27,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,283 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:27,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,284 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,287 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,287 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:27,287 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:27,287 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:27,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:27,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:27,287 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:27,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:27,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:27,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:27,287 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,287 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,287 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:27,287 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:27,287 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,287 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,287 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,387 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:27,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:27,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:27,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:27,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:27,391 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:27,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:27,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:27,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:27,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,392 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:27,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:27,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:27,392 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,393 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,393 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:27,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:27,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:27,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:27,394 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:27,394 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:27,394 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:27,394 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:27,394 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:27,394 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:27,394 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:27,394 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:27,394 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:27,394 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:27,394 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:27,394 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,394 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:27,394 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:27,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,396 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:27,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,396 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:27,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:27,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:27,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56d5aa26[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:27,398 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:27,398 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,400 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,400 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:27,400 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:27,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:27,400 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:27,401 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:27,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:27,403 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:27,403 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:27,403 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:27,403 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:27,403 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:27,405 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,405 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:27,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,405 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:27,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:27,405 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:27,406 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,409 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,409 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,409 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:27,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:27,409 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:27,409 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,409 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:27,409 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:27,409 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:27,409 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:27,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:27,409 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:27,409 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,409 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:27,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:27,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838987437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:27,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838987442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:27,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838987444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,509 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:27,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:27,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:27,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:27,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:27,512 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:27,512 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:27,512 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:27,512 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:27,513 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,513 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,515 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:27,515 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:27,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:27,516 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,516 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:27,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:27,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:27,517 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:27,517 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:27,517 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:27,517 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:27,517 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:27,518 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:27,518 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,518 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:27,518 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:27,519 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,519 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:27,519 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,519 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:27,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:27,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3077f0f5[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:27,521 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:27,521 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,522 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:27,522 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,522 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:27,522 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:27,524 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:27,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:27,526 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:27,526 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:27,526 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:27,526 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:27,526 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:27,529 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,529 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:27,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,529 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:27,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:27,529 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:27,530 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:27,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,535 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:27,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:27,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:27,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:27,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:27,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:27,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,535 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:27,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:27,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:27,535 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:27,569 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=340, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/5b0179092a6a43e9a9e0498e410f81b2 2023-03-26 13:55:27,577 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/c8d4c322ad8a48faaea247ebf842df00 2023-03-26 13:55:27,585 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/e042b3e28bf14fb4a9057a496b33ad57 2023-03-26 13:55:27,588 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/5b0179092a6a43e9a9e0498e410f81b2 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5b0179092a6a43e9a9e0498e410f81b2 2023-03-26 13:55:27,591 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5b0179092a6a43e9a9e0498e410f81b2, entries=150, sequenceid=340, filesize=25.4 K 2023-03-26 13:55:27,592 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/c8d4c322ad8a48faaea247ebf842df00 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/c8d4c322ad8a48faaea247ebf842df00 2023-03-26 13:55:27,594 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/c8d4c322ad8a48faaea247ebf842df00, entries=150, sequenceid=340, filesize=11.9 K 2023-03-26 13:55:27,595 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/e042b3e28bf14fb4a9057a496b33ad57 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/e042b3e28bf14fb4a9057a496b33ad57 2023-03-26 13:55:27,597 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/e042b3e28bf14fb4a9057a496b33ad57, entries=150, sequenceid=340, filesize=11.9 K 2023-03-26 13:55:27,598 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.42 KB/139690 for a9871eaedebb91319d17cf11612be1e0 in 855ms, sequenceid=340, compaction requested=true 2023-03-26 13:55:27,598 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:27,598 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:27,598 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:27,598 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:55:27,598 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:55:27,598 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:27,600 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104832 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:55:27,600 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:55:27,600 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a9871eaedebb91319d17cf11612be1e0/A is initiating minor compaction (all files) 2023-03-26 13:55:27,601 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a9871eaedebb91319d17cf11612be1e0/B is initiating minor compaction (all files) 2023-03-26 13:55:27,601 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a9871eaedebb91319d17cf11612be1e0/A in TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:27,601 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9871eaedebb91319d17cf11612be1e0/B in TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:27,601 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/2d0b2b733a10439aa353132584657a48, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/1f1351c60489466ab298f68ef352d701, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/1180852c24f84c9e839990eb75eb1b74, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5b0179092a6a43e9a9e0498e410f81b2] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp, totalSize=102.4 K 2023-03-26 13:55:27,601 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/2a3f9cc227334101b4685e3f99948904, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/d3bda2bbb3ff4d79b9f859f34b659956, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/e050f3e825ea468ca458f82d28432e40, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/c8d4c322ad8a48faaea247ebf842df00] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp, totalSize=48.2 K 2023-03-26 13:55:27,601 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 2d0b2b733a10439aa353132584657a48, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1679838923807 2023-03-26 13:55:27,601 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 2a3f9cc227334101b4685e3f99948904, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1679838923807 2023-03-26 13:55:27,601 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 1f1351c60489466ab298f68ef352d701, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679838923855 2023-03-26 13:55:27,601 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting d3bda2bbb3ff4d79b9f859f34b659956, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679838923855 2023-03-26 13:55:27,602 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 1180852c24f84c9e839990eb75eb1b74, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1679838924491 2023-03-26 13:55:27,602 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e050f3e825ea468ca458f82d28432e40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1679838924491 2023-03-26 13:55:27,602 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 5b0179092a6a43e9a9e0498e410f81b2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1679838926720 2023-03-26 13:55:27,602 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting c8d4c322ad8a48faaea247ebf842df00, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1679838926720 2023-03-26 13:55:27,610 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a9871eaedebb91319d17cf11612be1e0#B#compaction#421 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:27,610 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a9871eaedebb91319d17cf11612be1e0#A#compaction#420 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:27,636 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:27,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:27,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:27,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:27,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:27,639 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:27,639 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:27,639 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:27,639 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:27,640 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,640 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,641 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:27,641 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:27,641 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,641 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:27,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,642 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:27,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:27,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:27,642 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:27,642 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:27,642 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:27,642 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:27,642 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:27,643 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:27,643 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:27,643 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:27,643 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=136.42 KB heapSize=376 KB 2023-03-26 13:55:27,643 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:27,643 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:27,643 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:27,643 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:27,643 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:27,643 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:27,654 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326277574c95f4644118844b324721988a7 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326277574c95f4644118844b324721988a7 2023-03-26 13:55:27,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:27,952 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:27,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:27,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838987959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,970 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:27,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838987963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:27,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:27,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838987963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,029 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/65d71e2d88cc4cd383630ed3fd590516 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/65d71e2d88cc4cd383630ed3fd590516 2023-03-26 13:55:28,034 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a9871eaedebb91319d17cf11612be1e0/B of a9871eaedebb91319d17cf11612be1e0 into 65d71e2d88cc4cd383630ed3fd590516(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:28,034 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:28,034 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., storeName=a9871eaedebb91319d17cf11612be1e0/B, priority=12, startTime=1679838927598; duration=0sec 2023-03-26 13:55:28,034 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:28,034 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:55:28,036 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:55:28,036 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a9871eaedebb91319d17cf11612be1e0/C is initiating minor compaction (all files) 2023-03-26 13:55:28,037 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9871eaedebb91319d17cf11612be1e0/C in TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:28,037 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/738a41875770413b8d26b23a91a7f5e5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/eda66233b912405a8e461214b41f0e27, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/6fad7742ceef40c8949422571d96cb61, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/e042b3e28bf14fb4a9057a496b33ad57] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp, totalSize=48.2 K 2023-03-26 13:55:28,037 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 738a41875770413b8d26b23a91a7f5e5, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=291, earliestPutTs=1679838923807 2023-03-26 13:55:28,037 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting eda66233b912405a8e461214b41f0e27, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679838923855 2023-03-26 13:55:28,038 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 6fad7742ceef40c8949422571d96cb61, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1679838924491 2023-03-26 13:55:28,038 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e042b3e28bf14fb4a9057a496b33ad57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1679838926720 2023-03-26 13:55:28,045 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a9871eaedebb91319d17cf11612be1e0#C#compaction#423 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:28,059 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/1dbc0f92d285478ab4a1973e712b91d2 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/1dbc0f92d285478ab4a1973e712b91d2 2023-03-26 13:55:28,060 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=374, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/37d74aca3d944c44aada289544a24ee8 2023-03-26 13:55:28,066 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a9871eaedebb91319d17cf11612be1e0/C of a9871eaedebb91319d17cf11612be1e0 into 1dbc0f92d285478ab4a1973e712b91d2(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:28,066 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:28,067 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., storeName=a9871eaedebb91319d17cf11612be1e0/C, priority=12, startTime=1679838927598; duration=0sec 2023-03-26 13:55:28,067 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:28,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:28,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838988066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:28,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838988072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,077 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:28,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838988072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:28,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838988270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:28,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838988279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:28,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838988279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,445 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/9ac0e8ace5774b1da3f1522f4f6e5a07 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/9ac0e8ace5774b1da3f1522f4f6e5a07 2023-03-26 13:55:28,449 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a9871eaedebb91319d17cf11612be1e0/A of a9871eaedebb91319d17cf11612be1e0 into 9ac0e8ace5774b1da3f1522f4f6e5a07(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:28,449 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:28,449 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., storeName=a9871eaedebb91319d17cf11612be1e0/A, priority=12, startTime=1679838927598; duration=0sec 2023-03-26 13:55:28,449 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:28,481 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/93c963a6544c439f9c6be7f7ac694490 2023-03-26 13:55:28,491 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/c5b889f10a0f404cab9f43af96198ce8 2023-03-26 13:55:28,494 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/37d74aca3d944c44aada289544a24ee8 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/37d74aca3d944c44aada289544a24ee8 2023-03-26 13:55:28,498 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/37d74aca3d944c44aada289544a24ee8, entries=150, sequenceid=374, filesize=25.4 K 2023-03-26 13:55:28,499 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/93c963a6544c439f9c6be7f7ac694490 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/93c963a6544c439f9c6be7f7ac694490 2023-03-26 13:55:28,502 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/93c963a6544c439f9c6be7f7ac694490, entries=150, sequenceid=374, filesize=11.9 K 2023-03-26 13:55:28,502 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/c5b889f10a0f404cab9f43af96198ce8 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/c5b889f10a0f404cab9f43af96198ce8 2023-03-26 13:55:28,525 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/c5b889f10a0f404cab9f43af96198ce8, entries=150, sequenceid=374, filesize=11.9 K 2023-03-26 13:55:28,526 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~375.95 KB/384976, currentSize=46.96 KB/48090 for a9871eaedebb91319d17cf11612be1e0 in 883ms, sequenceid=374, compaction requested=false 2023-03-26 13:55:28,526 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:28,526 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:28,526 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:28,526 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:28,526 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,526 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:28,526 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:28,528 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,528 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:28,528 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,528 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:28,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:28,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@452ec8e3[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:28,530 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:28,530 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,535 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:28,535 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:28,535 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:28,535 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,535 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:28,536 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,536 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:28,536 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:28,537 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:28,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:28,539 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:28,539 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:28,539 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:28,539 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:28,539 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:28,540 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:28,540 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,540 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:28,541 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,542 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,544 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:28,544 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:28,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:28,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:28,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:28,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:28,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:28,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:28,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:28,545 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:28,544 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,544 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:28,545 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:28,545 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:28,545 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:28,545 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,545 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,545 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,545 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:28,597 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 13:55:28,597 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:28,597 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:28,597 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:28,597 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:28,597 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:28,597 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:28,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838928622, max timestamp=1679838928632Num uniques 859; 2023-03-26 13:55:28,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:55:28,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838928622, max timestamp=1679838928632Num uniques 859; 2023-03-26 13:55:28,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:55:28,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838928622, max timestamp=1679838928632Num uniques 859; 2023-03-26 13:55:28,634 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:55:28,645 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:28,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:28,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:28,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:28,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:28,648 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:28,648 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:28,648 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:28,648 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:28,649 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,649 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:28,651 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,651 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:28,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:28,651 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,651 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:28,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:28,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:28,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:28,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:28,652 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:28,652 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:28,652 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:28,653 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:28,653 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:28,653 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:28,653 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:28,653 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:28,653 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:28,653 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:28,653 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:28,653 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,653 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:28,653 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:28,654 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,654 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:28,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:28,655 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,655 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:28,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:28,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:28,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:28,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:28,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23a776e1[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:28,656 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:28,656 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,657 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,657 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:28,657 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:28,657 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:28,657 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,657 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:28,660 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,660 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:28,660 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:28,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:28,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:28,660 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:28,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:28,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:28,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:28,662 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:28,662 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:28,663 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:28,663 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:28,663 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:28,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:28,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838988655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:28,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838988661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,665 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,665 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,665 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:28,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:28,665 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:28,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:28,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,666 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:28,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:28,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838988662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,668 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,668 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:28,668 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:28,668 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:28,669 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:28,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:28,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:28,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:28,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:28,668 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:28,669 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,669 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:28,669 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:28,669 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:28,669 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,669 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:28,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:28,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:28,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838988765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:28,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838988765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,769 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:28,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:28,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:28,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838988769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:28,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:28,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:28,774 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:28,774 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:28,774 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:28,774 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:28,774 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,774 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:28,775 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:28,775 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:28,776 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,776 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:28,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:28,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:28,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:28,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:28,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:28,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:28,777 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:28,777 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:28,777 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:28,779 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:28,779 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:28,779 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:28,779 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:28,779 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:28,779 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:28,779 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:28,779 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:28,779 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,780 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:28,780 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:28,781 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,781 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:28,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:28,781 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,781 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:28,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:28,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:28,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:28,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:28,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58586c29[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:28,783 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:28,783 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,785 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,786 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:28,786 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,786 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:28,786 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:28,786 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:28,787 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,787 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:28,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:28,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:28,787 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:28,787 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:28,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:28,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:28,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:28,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:28,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:28,790 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:28,790 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:28,790 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:28,790 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:28,790 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:28,791 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,791 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,791 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:28,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:28,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:28,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:28,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:28,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:28,792 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:28,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,793 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:28,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,799 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:28,799 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,799 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:28,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:28,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:28,799 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:28,799 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:28,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:28,799 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:28,799 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:28,799 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:28,799 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,799 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:28,799 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,800 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,899 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:28,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:28,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:28,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:28,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:28,903 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:28,903 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:28,903 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:28,903 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:28,903 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,903 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:28,904 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,904 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:28,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:28,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:28,904 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,905 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:28,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:28,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:28,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:28,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:28,905 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:28,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:28,906 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:28,906 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:28,906 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:28,906 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:28,906 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:28,906 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:28,906 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:28,906 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:28,906 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:28,906 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:28,906 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:28,906 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,906 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:28,906 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:28,907 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,907 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:28,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:28,908 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,908 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:28,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:28,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:28,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:28,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:28,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5727cf8f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:28,909 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:28,909 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,911 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,912 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:28,912 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:28,912 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:28,912 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,912 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:28,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,913 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:28,913 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:28,913 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:28,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:28,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:28,915 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:28,915 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:28,915 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:28,915 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:28,915 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:28,916 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,916 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:28,917 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:28,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:28,917 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,918 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:28,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:28,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,920 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:28,920 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:28,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:28,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:28,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:28,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:28,921 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,921 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:28,921 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:28,921 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,921 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,921 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:28,922 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:28,922 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:28,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:28,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:28,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:28,922 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:28,922 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:28,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:28,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838988970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:28,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838988972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:28,976 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:28,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838988972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,007 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326803f5ea02daf40fba0dd03ef1290f62f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326803f5ea02daf40fba0dd03ef1290f62f 2023-03-26 13:55:29,022 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:29,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:29,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:29,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:29,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:29,025 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:29,025 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:29,025 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:29,025 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:29,025 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,026 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,028 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,028 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:29,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:29,028 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,028 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:29,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,029 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:29,029 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:29,029 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:29,029 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:29,029 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:29,029 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:29,030 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:29,030 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:29,030 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:29,030 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:29,030 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:29,031 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,031 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:29,032 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:29,032 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:29,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:29,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64fd6c58[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:29,034 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:29,034 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,035 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,035 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:29,035 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:29,035 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:29,035 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,035 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:29,037 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:29,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:29,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:29,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:29,039 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:29,039 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:29,039 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:29,039 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:29,039 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:29,042 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,042 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,042 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:29,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:29,042 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:29,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:29,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:29,043 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:29,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,046 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:29,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:29,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:29,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:29,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:29,046 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:29,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:29,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:29,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:29,046 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:29,046 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:29,147 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:29,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:29,148 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:29,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:29,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:29,149 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:29,149 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:29,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:29,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:29,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,151 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:29,151 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:29,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:29,151 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,151 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:29,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:29,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:29,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:29,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:29,152 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:29,152 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:29,152 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:29,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:29,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:29,153 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:29,153 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:29,153 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:29,153 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:29,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:29,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:29,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:29,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:29,154 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,154 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:29,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:29,155 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,155 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:29,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:29,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:29,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cf50e58[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:29,157 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:29,157 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,159 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,159 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:29,159 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:29,159 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:29,159 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,159 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:29,161 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,161 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:29,161 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:29,161 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:29,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:29,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:29,163 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:29,163 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:29,163 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:29,163 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:29,163 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:29,165 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,165 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,165 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:29,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:29,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:29,165 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:29,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,167 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,169 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:29,169 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:29,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:29,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:29,169 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:29,169 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:29,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:29,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:29,169 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,169 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,169 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,169 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,170 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,170 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,170 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:29,170 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:29,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:29,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:29,270 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:29,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:29,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:29,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:29,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:29,273 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:29,273 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:29,273 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:29,273 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:29,273 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,273 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:29,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838989274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,279 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:29,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838989276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:29,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:29,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:29,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:29,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:29,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:29,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:29,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:29,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:29,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:29,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:29,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:29,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:29,282 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:29,282 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:29,282 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:29,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:29,282 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:29,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838989278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,283 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:29,283 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:29,283 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,283 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:29,283 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:29,284 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:29,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:29,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:29,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:29,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:29,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:29,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:29,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67571ccb[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:29,286 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:29,286 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,288 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,288 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,288 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:29,288 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:29,288 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:29,288 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:29,289 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,289 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:29,289 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:29,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:29,290 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:29,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:29,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:29,292 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:29,292 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:29,292 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:29,292 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:29,292 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:29,293 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,293 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,293 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:29,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:29,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:29,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:29,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:29,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:29,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:29,295 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:29,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,296 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,299 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,299 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,299 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,299 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:29,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:29,299 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:29,299 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,299 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:29,299 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:29,299 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:29,299 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:29,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:29,299 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:29,299 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:29,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:29,399 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:29,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:29,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:29,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:29,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:29,403 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:29,403 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:29,403 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:29,403 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:29,403 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,403 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,409 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,409 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:29,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:29,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:29,409 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,410 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:29,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:29,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:29,411 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:29,411 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:29,411 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:29,411 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:29,411 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:29,411 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:29,411 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. started... 2023-03-26 13:55:29,411 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. as already flushing 2023-03-26 13:55:29,411 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:29,411 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:29,411 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:29,411 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,411 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:29,411 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:29,412 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,412 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:29,413 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,413 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,414 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=388, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/14dcaabfb7424c2fb7edfa5bbb8e9ff3 2023-03-26 13:55:29,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:29,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:29,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ee5ec19[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:29,414 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:29,415 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,416 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,416 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:29,416 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:29,416 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:29,416 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,416 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:29,417 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,417 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:29,417 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:29,418 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:29,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:29,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:29,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:29,420 DEBUG [Thread-2688] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b73a361 to 127.0.0.1:54332 2023-03-26 13:55:29,420 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:29,420 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:29,420 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:29,420 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:29,420 DEBUG [Thread-2680] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x678b4c3d to 127.0.0.1:54332 2023-03-26 13:55:29,420 DEBUG [Thread-2688] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:55:29,420 DEBUG [Thread-2680] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:55:29,420 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:29,423 DEBUG [Thread-2682] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43736f95 to 127.0.0.1:54332 2023-03-26 13:55:29,423 DEBUG [Thread-2682] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:55:29,426 DEBUG [Thread-2684] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x72eaaf6e to 127.0.0.1:54332 2023-03-26 13:55:29,426 DEBUG [Thread-2684] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:55:29,427 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,427 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,427 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:29,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,427 DEBUG [Thread-2686] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4130667f to 127.0.0.1:54332 2023-03-26 13:55:29,427 DEBUG [Thread-2686] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:55:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:29,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:29,434 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,435 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:29,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:29,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:29,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:29,437 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:29,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:29,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:29,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:29,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:29,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:29,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:29,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,437 DEBUG [Thread-2679] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:29,437 DEBUG [Thread-2679] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:29,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:29,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:29,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:29,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:29,538 DEBUG [Thread-2679] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:29,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:29,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:29,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53846 deadline: 1679838989783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:29,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53826 deadline: 1679838989783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:29,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53818 deadline: 1679838989784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,825 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/8c01a2fa975344938918124e01fed644 2023-03-26 13:55:29,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:29,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53794 deadline: 1679838989962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:29,964 DEBUG [Thread-2677] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18220 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., hostname=jenkins-hbase4.apache.org,42517,1679838814721, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:55:30,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:30,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53804 deadline: 1679838990013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:30,015 DEBUG [Thread-2669] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18281 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., hostname=jenkins-hbase4.apache.org,42517,1679838814721, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:55:30,234 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/b16d40149770429d9be292a0c4171c31 2023-03-26 13:55:30,237 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/14dcaabfb7424c2fb7edfa5bbb8e9ff3 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/14dcaabfb7424c2fb7edfa5bbb8e9ff3 2023-03-26 13:55:30,239 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/14dcaabfb7424c2fb7edfa5bbb8e9ff3, entries=150, sequenceid=388, filesize=25.4 K 2023-03-26 13:55:30,239 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/8c01a2fa975344938918124e01fed644 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/8c01a2fa975344938918124e01fed644 2023-03-26 13:55:30,242 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/8c01a2fa975344938918124e01fed644, entries=100, sequenceid=388, filesize=9.5 K 2023-03-26 13:55:30,242 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/b16d40149770429d9be292a0c4171c31 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/b16d40149770429d9be292a0c4171c31 2023-03-26 13:55:30,245 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/b16d40149770429d9be292a0c4171c31, entries=100, sequenceid=388, filesize=9.5 K 2023-03-26 13:55:30,245 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for a9871eaedebb91319d17cf11612be1e0 in 1648ms, sequenceid=388, compaction requested=true 2023-03-26 13:55:30,245 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:30,245 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:30,245 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:30,245 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:30,245 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:30,245 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:30,246 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78918 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:30,246 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34898 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:30,246 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): a9871eaedebb91319d17cf11612be1e0/A is initiating minor compaction (all files) 2023-03-26 13:55:30,246 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a9871eaedebb91319d17cf11612be1e0/B is initiating minor compaction (all files) 2023-03-26 13:55:30,246 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a9871eaedebb91319d17cf11612be1e0/A in TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:30,246 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9871eaedebb91319d17cf11612be1e0/B in TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:30,246 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/9ac0e8ace5774b1da3f1522f4f6e5a07, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/37d74aca3d944c44aada289544a24ee8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/14dcaabfb7424c2fb7edfa5bbb8e9ff3] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp, totalSize=77.1 K 2023-03-26 13:55:30,246 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/65d71e2d88cc4cd383630ed3fd590516, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/93c963a6544c439f9c6be7f7ac694490, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/8c01a2fa975344938918124e01fed644] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp, totalSize=34.1 K 2023-03-26 13:55:30,247 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 9ac0e8ace5774b1da3f1522f4f6e5a07, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1679838926720 2023-03-26 13:55:30,247 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 65d71e2d88cc4cd383630ed3fd590516, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1679838926720 2023-03-26 13:55:30,247 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 37d74aca3d944c44aada289544a24ee8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1679838926806 2023-03-26 13:55:30,247 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 93c963a6544c439f9c6be7f7ac694490, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1679838926806 2023-03-26 13:55:30,247 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 14dcaabfb7424c2fb7edfa5bbb8e9ff3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1679838927958 2023-03-26 13:55:30,247 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 8c01a2fa975344938918124e01fed644, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1679838927958 2023-03-26 13:55:30,252 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a9871eaedebb91319d17cf11612be1e0#B#compaction#430 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:30,260 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): a9871eaedebb91319d17cf11612be1e0#A#compaction#429 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:30,665 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/718f66eb291f43e19abbbcc1b1ff70c6 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/718f66eb291f43e19abbbcc1b1ff70c6 2023-03-26 13:55:30,669 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9871eaedebb91319d17cf11612be1e0/B of a9871eaedebb91319d17cf11612be1e0 into 718f66eb291f43e19abbbcc1b1ff70c6(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:30,669 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:30,669 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., storeName=a9871eaedebb91319d17cf11612be1e0/B, priority=13, startTime=1679838930245; duration=0sec 2023-03-26 13:55:30,669 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:30,669 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:30,670 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34898 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:30,670 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): a9871eaedebb91319d17cf11612be1e0/C is initiating minor compaction (all files) 2023-03-26 13:55:30,670 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a9871eaedebb91319d17cf11612be1e0/C in TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:30,670 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/1dbc0f92d285478ab4a1973e712b91d2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/c5b889f10a0f404cab9f43af96198ce8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/b16d40149770429d9be292a0c4171c31] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp, totalSize=34.1 K 2023-03-26 13:55:30,670 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 1dbc0f92d285478ab4a1973e712b91d2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1679838926720 2023-03-26 13:55:30,671 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting c5b889f10a0f404cab9f43af96198ce8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1679838926806 2023-03-26 13:55:30,671 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting b16d40149770429d9be292a0c4171c31, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1679838927958 2023-03-26 13:55:30,675 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): a9871eaedebb91319d17cf11612be1e0#C#compaction#431 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:30,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:30,792 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-26 13:55:30,792 DEBUG [Thread-2671] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2329054b to 127.0.0.1:54332 2023-03-26 13:55:30,792 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:30,792 DEBUG [Thread-2675] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5baa6768 to 127.0.0.1:54332 2023-03-26 13:55:30,792 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:30,792 DEBUG [Thread-2675] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:55:30,792 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:30,792 DEBUG [Thread-2671] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:55:30,792 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:30,793 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:30,793 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:30,793 DEBUG [Thread-2673] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30951255 to 127.0.0.1:54332 2023-03-26 13:55:30,793 DEBUG [Thread-2673] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:55:31,073 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/2e7883797f5440049e2be9497bfc37ef as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/2e7883797f5440049e2be9497bfc37ef 2023-03-26 13:55:31,076 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9871eaedebb91319d17cf11612be1e0/A of a9871eaedebb91319d17cf11612be1e0 into 2e7883797f5440049e2be9497bfc37ef(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:31,076 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:31,076 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., storeName=a9871eaedebb91319d17cf11612be1e0/A, priority=13, startTime=1679838930245; duration=0sec 2023-03-26 13:55:31,076 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:31,081 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/5bb19783e2d2432cb8fa15c5a467d98b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/5bb19783e2d2432cb8fa15c5a467d98b 2023-03-26 13:55:31,084 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a9871eaedebb91319d17cf11612be1e0/C of a9871eaedebb91319d17cf11612be1e0 into 5bb19783e2d2432cb8fa15c5a467d98b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:31,084 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:31,084 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0., storeName=a9871eaedebb91319d17cf11612be1e0/C, priority=13, startTime=1679838930245; duration=0sec 2023-03-26 13:55:31,084 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:31,203 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326091534583a7a4c849b1e425fbdc13778 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326091534583a7a4c849b1e425fbdc13778 2023-03-26 13:55:31,607 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=426, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/ba8e924755da4e9f95934ab4f5f5a93a 2023-03-26 13:55:32,015 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/384d93c3e2a84f7e8c40ed7659685f5c 2023-03-26 13:55:32,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/0499e46b49ff40fe8e977c2f66bb6872 2023-03-26 13:55:32,437 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/ba8e924755da4e9f95934ab4f5f5a93a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/ba8e924755da4e9f95934ab4f5f5a93a 2023-03-26 13:55:32,439 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/ba8e924755da4e9f95934ab4f5f5a93a, entries=150, sequenceid=426, filesize=25.4 K 2023-03-26 13:55:32,440 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/384d93c3e2a84f7e8c40ed7659685f5c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/384d93c3e2a84f7e8c40ed7659685f5c 2023-03-26 13:55:32,443 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/384d93c3e2a84f7e8c40ed7659685f5c, entries=150, sequenceid=426, filesize=11.9 K 2023-03-26 13:55:32,443 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/0499e46b49ff40fe8e977c2f66bb6872 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/0499e46b49ff40fe8e977c2f66bb6872 2023-03-26 13:55:32,446 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/0499e46b49ff40fe8e977c2f66bb6872, entries=150, sequenceid=426, filesize=11.9 K 2023-03-26 13:55:32,447 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=6.71 KB/6870 for a9871eaedebb91319d17cf11612be1e0 in 1655ms, sequenceid=426, compaction requested=false 2023-03-26 13:55:32,447 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:36,094 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-26 13:55:36,126 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5262cda5f8fa47bc874a077bb9d8d2b6, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/fd97c05aaf5446da983dfb9bf969028f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/4a5a4ebfcb1b43a991d0ffa498dd46c3, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/8980a7aa5ebc4dbd8d38d1d2dfca2b76, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5ee591eb493f42458f367162d572f9f8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/8ff872b31fa74f24b3a38176c6162b1a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/f21b3ba02e0c4d12a7b38cc98ff3b7d8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/01b7d924b1fb4de2b340be18c61d73f5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/0437d49d2c2e4bc09c378c1c0ceb9646, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/12fdc39eff514e01bf482ee463d5b692, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5989223eab014cbdb1a33966a307f891, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/3b10b87c8599410a846f708a8fb58650, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/29c97c21827f4d2ea89a4a3d30c33460, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/e3a2b1e18471428c8f280078d383e45d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/b0ca98eb4c7440f29cc1ce72ec7ca083, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/d25dc7673a064187b494ea181160b257, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/e76c673de3f2468a956837ff2c57acdc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/df750aec4ac849ef91e1dd7f424848f5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/f8cda67799a24e668a2efec2d39f088f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/2d0b2b733a10439aa353132584657a48, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/1f1351c60489466ab298f68ef352d701, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/1180852c24f84c9e839990eb75eb1b74, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/9ac0e8ace5774b1da3f1522f4f6e5a07, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5b0179092a6a43e9a9e0498e410f81b2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/37d74aca3d944c44aada289544a24ee8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/14dcaabfb7424c2fb7edfa5bbb8e9ff3] to archive 2023-03-26 13:55:36,127 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 13:55:36,129 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5262cda5f8fa47bc874a077bb9d8d2b6 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5262cda5f8fa47bc874a077bb9d8d2b6 2023-03-26 13:55:36,129 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/fd97c05aaf5446da983dfb9bf969028f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/fd97c05aaf5446da983dfb9bf969028f 2023-03-26 13:55:36,130 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/4a5a4ebfcb1b43a991d0ffa498dd46c3 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/4a5a4ebfcb1b43a991d0ffa498dd46c3 2023-03-26 13:55:36,131 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/8980a7aa5ebc4dbd8d38d1d2dfca2b76 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/8980a7aa5ebc4dbd8d38d1d2dfca2b76 2023-03-26 13:55:36,132 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5ee591eb493f42458f367162d572f9f8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5ee591eb493f42458f367162d572f9f8 2023-03-26 13:55:36,133 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/8ff872b31fa74f24b3a38176c6162b1a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/8ff872b31fa74f24b3a38176c6162b1a 2023-03-26 13:55:36,134 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/f21b3ba02e0c4d12a7b38cc98ff3b7d8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/f21b3ba02e0c4d12a7b38cc98ff3b7d8 2023-03-26 13:55:36,135 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/01b7d924b1fb4de2b340be18c61d73f5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/01b7d924b1fb4de2b340be18c61d73f5 2023-03-26 13:55:36,136 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/0437d49d2c2e4bc09c378c1c0ceb9646 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/0437d49d2c2e4bc09c378c1c0ceb9646 2023-03-26 13:55:36,137 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/12fdc39eff514e01bf482ee463d5b692 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/12fdc39eff514e01bf482ee463d5b692 2023-03-26 13:55:36,138 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5989223eab014cbdb1a33966a307f891 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5989223eab014cbdb1a33966a307f891 2023-03-26 13:55:36,138 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/3b10b87c8599410a846f708a8fb58650 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/3b10b87c8599410a846f708a8fb58650 2023-03-26 13:55:36,139 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/29c97c21827f4d2ea89a4a3d30c33460 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/29c97c21827f4d2ea89a4a3d30c33460 2023-03-26 13:55:36,140 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/e3a2b1e18471428c8f280078d383e45d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/e3a2b1e18471428c8f280078d383e45d 2023-03-26 13:55:36,141 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/b0ca98eb4c7440f29cc1ce72ec7ca083 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/b0ca98eb4c7440f29cc1ce72ec7ca083 2023-03-26 13:55:36,141 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/d25dc7673a064187b494ea181160b257 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/d25dc7673a064187b494ea181160b257 2023-03-26 13:55:36,142 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/e76c673de3f2468a956837ff2c57acdc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/e76c673de3f2468a956837ff2c57acdc 2023-03-26 13:55:36,143 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/df750aec4ac849ef91e1dd7f424848f5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/df750aec4ac849ef91e1dd7f424848f5 2023-03-26 13:55:36,144 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/f8cda67799a24e668a2efec2d39f088f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/f8cda67799a24e668a2efec2d39f088f 2023-03-26 13:55:36,145 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/2d0b2b733a10439aa353132584657a48 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/2d0b2b733a10439aa353132584657a48 2023-03-26 13:55:36,145 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/1f1351c60489466ab298f68ef352d701 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/1f1351c60489466ab298f68ef352d701 2023-03-26 13:55:36,146 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/1180852c24f84c9e839990eb75eb1b74 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/1180852c24f84c9e839990eb75eb1b74 2023-03-26 13:55:36,147 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/9ac0e8ace5774b1da3f1522f4f6e5a07 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/9ac0e8ace5774b1da3f1522f4f6e5a07 2023-03-26 13:55:36,148 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5b0179092a6a43e9a9e0498e410f81b2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5b0179092a6a43e9a9e0498e410f81b2 2023-03-26 13:55:36,148 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/37d74aca3d944c44aada289544a24ee8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/37d74aca3d944c44aada289544a24ee8 2023-03-26 13:55:36,149 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/14dcaabfb7424c2fb7edfa5bbb8e9ff3 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/14dcaabfb7424c2fb7edfa5bbb8e9ff3 2023-03-26 13:55:36,153 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/371e209d940147d692de96e5d483aa9f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/5bdb5ef59a8b4309b388d5a66ff8f97a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/dcfb160bfa4e47d58fb8acdbfc33eb86, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/4cdcf89ed49a4db4880f28ea8e5d02a3, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/12708e34f8744bfe94f52e8e2cc412d9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/cdbdd69f04e14654a50892575667f086, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/36225ca64c274c45b13c3ff75d33f640, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/ec196239ef104244aacfe16f0945cfd4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/cbe17c6cf273476cb8a4252beae59b99, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/515e6f451db846d8bb73ceb384c7b202, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/f3e527038832454dab9be1b9d4b0a85c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/c0c3d832cc124f5689c21c97010ae868, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/82d03ddcd7dc45de8e5b044818ad6f44, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/c09d0d50f46941cd9f12f3779ef15e2e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/0ca0c294676c49969508c2eae85999ae, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/48b33fd5bca5466d949c8a3b7d112131, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/70192155db694a76b032e983888e5429, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/541f0ee4985b4c9cbbfa5ac1a434782b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/2a3f9cc227334101b4685e3f99948904, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/0cae603fad67442c8bc0cfba7228f593, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/d3bda2bbb3ff4d79b9f859f34b659956, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/e050f3e825ea468ca458f82d28432e40, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/65d71e2d88cc4cd383630ed3fd590516, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/c8d4c322ad8a48faaea247ebf842df00, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/93c963a6544c439f9c6be7f7ac694490, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/8c01a2fa975344938918124e01fed644] to archive 2023-03-26 13:55:36,154 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 13:55:36,155 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/371e209d940147d692de96e5d483aa9f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/371e209d940147d692de96e5d483aa9f 2023-03-26 13:55:36,156 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/5bdb5ef59a8b4309b388d5a66ff8f97a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/5bdb5ef59a8b4309b388d5a66ff8f97a 2023-03-26 13:55:36,156 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/dcfb160bfa4e47d58fb8acdbfc33eb86 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/dcfb160bfa4e47d58fb8acdbfc33eb86 2023-03-26 13:55:36,157 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/4cdcf89ed49a4db4880f28ea8e5d02a3 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/4cdcf89ed49a4db4880f28ea8e5d02a3 2023-03-26 13:55:36,158 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/12708e34f8744bfe94f52e8e2cc412d9 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/12708e34f8744bfe94f52e8e2cc412d9 2023-03-26 13:55:36,158 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/cdbdd69f04e14654a50892575667f086 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/cdbdd69f04e14654a50892575667f086 2023-03-26 13:55:36,159 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/36225ca64c274c45b13c3ff75d33f640 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/36225ca64c274c45b13c3ff75d33f640 2023-03-26 13:55:36,159 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/ec196239ef104244aacfe16f0945cfd4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/ec196239ef104244aacfe16f0945cfd4 2023-03-26 13:55:36,160 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/cbe17c6cf273476cb8a4252beae59b99 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/cbe17c6cf273476cb8a4252beae59b99 2023-03-26 13:55:36,161 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/515e6f451db846d8bb73ceb384c7b202 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/515e6f451db846d8bb73ceb384c7b202 2023-03-26 13:55:36,161 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/f3e527038832454dab9be1b9d4b0a85c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/f3e527038832454dab9be1b9d4b0a85c 2023-03-26 13:55:36,162 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/c0c3d832cc124f5689c21c97010ae868 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/c0c3d832cc124f5689c21c97010ae868 2023-03-26 13:55:36,163 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/82d03ddcd7dc45de8e5b044818ad6f44 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/82d03ddcd7dc45de8e5b044818ad6f44 2023-03-26 13:55:36,163 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/c09d0d50f46941cd9f12f3779ef15e2e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/c09d0d50f46941cd9f12f3779ef15e2e 2023-03-26 13:55:36,164 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/0ca0c294676c49969508c2eae85999ae to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/0ca0c294676c49969508c2eae85999ae 2023-03-26 13:55:36,165 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/48b33fd5bca5466d949c8a3b7d112131 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/48b33fd5bca5466d949c8a3b7d112131 2023-03-26 13:55:36,166 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/70192155db694a76b032e983888e5429 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/70192155db694a76b032e983888e5429 2023-03-26 13:55:36,166 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/541f0ee4985b4c9cbbfa5ac1a434782b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/541f0ee4985b4c9cbbfa5ac1a434782b 2023-03-26 13:55:36,167 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/2a3f9cc227334101b4685e3f99948904 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/2a3f9cc227334101b4685e3f99948904 2023-03-26 13:55:36,168 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/0cae603fad67442c8bc0cfba7228f593 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/0cae603fad67442c8bc0cfba7228f593 2023-03-26 13:55:36,168 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/d3bda2bbb3ff4d79b9f859f34b659956 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/d3bda2bbb3ff4d79b9f859f34b659956 2023-03-26 13:55:36,169 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/e050f3e825ea468ca458f82d28432e40 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/e050f3e825ea468ca458f82d28432e40 2023-03-26 13:55:36,170 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/65d71e2d88cc4cd383630ed3fd590516 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/65d71e2d88cc4cd383630ed3fd590516 2023-03-26 13:55:36,170 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/c8d4c322ad8a48faaea247ebf842df00 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/c8d4c322ad8a48faaea247ebf842df00 2023-03-26 13:55:36,171 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/93c963a6544c439f9c6be7f7ac694490 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/93c963a6544c439f9c6be7f7ac694490 2023-03-26 13:55:36,172 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/8c01a2fa975344938918124e01fed644 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/8c01a2fa975344938918124e01fed644 2023-03-26 13:55:36,176 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/672ce41860874484b85de7444618af7c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/2822f467167142cf9e48d96f7b54001b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/12317b30bb5b4190a3c16c0678f07381, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/62f5cd7dc2354cde9befe212ed428027, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/2955b3869afb48ae87bc6b0020fec528, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/f378b8b6ea604720b75114399539156a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/9a9d5468e7954a13b61a7b1fdd46e481, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/1e823dee8e8b460db53be47f09b4e16c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/1853a75ba65d42de8604d30f0cd5b31b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/54248848bbac4caa8f61092bc8b4ecf7, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/28959ac0406f47a5bc2e7af6274c3433, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/3e3ff27aff484d0494be6293e74c52ca, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/3b2876f4916f45f7a1c5861894903913, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/dc2d9621b6684c85b6f885b31b2ad9aa, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/be4253f0533a4d3fb8fbc154e9100cb4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/34fa8e8e75f64c38907c9499f565fa20, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/7c75ddea9c64411aa23fe0d596f840f8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/143dc90fc6c6436d9f4042ef83fc185a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/738a41875770413b8d26b23a91a7f5e5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/9b63d39fadf8403eab6212ec8079cd8f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/eda66233b912405a8e461214b41f0e27, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/6fad7742ceef40c8949422571d96cb61, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/1dbc0f92d285478ab4a1973e712b91d2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/e042b3e28bf14fb4a9057a496b33ad57, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/c5b889f10a0f404cab9f43af96198ce8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/b16d40149770429d9be292a0c4171c31] to archive 2023-03-26 13:55:36,177 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 13:55:36,178 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/672ce41860874484b85de7444618af7c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/672ce41860874484b85de7444618af7c 2023-03-26 13:55:36,178 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/2822f467167142cf9e48d96f7b54001b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/2822f467167142cf9e48d96f7b54001b 2023-03-26 13:55:36,179 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/12317b30bb5b4190a3c16c0678f07381 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/12317b30bb5b4190a3c16c0678f07381 2023-03-26 13:55:36,180 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/62f5cd7dc2354cde9befe212ed428027 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/62f5cd7dc2354cde9befe212ed428027 2023-03-26 13:55:36,181 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/2955b3869afb48ae87bc6b0020fec528 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/2955b3869afb48ae87bc6b0020fec528 2023-03-26 13:55:36,182 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/f378b8b6ea604720b75114399539156a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/f378b8b6ea604720b75114399539156a 2023-03-26 13:55:36,182 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/9a9d5468e7954a13b61a7b1fdd46e481 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/9a9d5468e7954a13b61a7b1fdd46e481 2023-03-26 13:55:36,183 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/1e823dee8e8b460db53be47f09b4e16c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/1e823dee8e8b460db53be47f09b4e16c 2023-03-26 13:55:36,184 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/1853a75ba65d42de8604d30f0cd5b31b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/1853a75ba65d42de8604d30f0cd5b31b 2023-03-26 13:55:36,185 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/54248848bbac4caa8f61092bc8b4ecf7 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/54248848bbac4caa8f61092bc8b4ecf7 2023-03-26 13:55:36,185 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/28959ac0406f47a5bc2e7af6274c3433 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/28959ac0406f47a5bc2e7af6274c3433 2023-03-26 13:55:36,186 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/3e3ff27aff484d0494be6293e74c52ca to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/3e3ff27aff484d0494be6293e74c52ca 2023-03-26 13:55:36,187 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/3b2876f4916f45f7a1c5861894903913 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/3b2876f4916f45f7a1c5861894903913 2023-03-26 13:55:36,188 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/dc2d9621b6684c85b6f885b31b2ad9aa to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/dc2d9621b6684c85b6f885b31b2ad9aa 2023-03-26 13:55:36,188 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/be4253f0533a4d3fb8fbc154e9100cb4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/be4253f0533a4d3fb8fbc154e9100cb4 2023-03-26 13:55:36,189 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/34fa8e8e75f64c38907c9499f565fa20 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/34fa8e8e75f64c38907c9499f565fa20 2023-03-26 13:55:36,190 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/7c75ddea9c64411aa23fe0d596f840f8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/7c75ddea9c64411aa23fe0d596f840f8 2023-03-26 13:55:36,191 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/143dc90fc6c6436d9f4042ef83fc185a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/143dc90fc6c6436d9f4042ef83fc185a 2023-03-26 13:55:36,191 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/738a41875770413b8d26b23a91a7f5e5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/738a41875770413b8d26b23a91a7f5e5 2023-03-26 13:55:36,192 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/9b63d39fadf8403eab6212ec8079cd8f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/9b63d39fadf8403eab6212ec8079cd8f 2023-03-26 13:55:36,193 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/eda66233b912405a8e461214b41f0e27 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/eda66233b912405a8e461214b41f0e27 2023-03-26 13:55:36,194 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/6fad7742ceef40c8949422571d96cb61 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/6fad7742ceef40c8949422571d96cb61 2023-03-26 13:55:36,195 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/1dbc0f92d285478ab4a1973e712b91d2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/1dbc0f92d285478ab4a1973e712b91d2 2023-03-26 13:55:36,196 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/e042b3e28bf14fb4a9057a496b33ad57 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/e042b3e28bf14fb4a9057a496b33ad57 2023-03-26 13:55:36,197 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/c5b889f10a0f404cab9f43af96198ce8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/c5b889f10a0f404cab9f43af96198ce8 2023-03-26 13:55:36,198 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/b16d40149770429d9be292a0c4171c31 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/b16d40149770429d9be292a0c4171c31 2023-03-26 13:55:40,028 DEBUG [Thread-2677] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63a535f3 to 127.0.0.1:54332 2023-03-26 13:55:40,028 DEBUG [Thread-2677] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:55:40,066 DEBUG [Thread-2669] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d800352 to 127.0.0.1:54332 2023-03-26 13:55:40,066 DEBUG [Thread-2669] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:55:40,066 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-26 13:55:40,067 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 17 2023-03-26 13:55:40,067 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 99 2023-03-26 13:55:40,067 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 103 2023-03-26 13:55:40,067 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 100 2023-03-26 13:55:40,067 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 19 2023-03-26 13:55:40,067 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-26 13:55:40,067 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-26 13:55:40,067 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(402): scanned 1836 2023-03-26 13:55:40,067 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(403): verified 5497 rows 2023-03-26 13:55:40,067 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(402): scanned 1822 2023-03-26 13:55:40,067 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(403): verified 5461 rows 2023-03-26 13:55:40,067 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(402): scanned 1821 2023-03-26 13:55:40,067 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(403): verified 5461 rows 2023-03-26 13:55:40,067 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(402): scanned 1799 2023-03-26 13:55:40,067 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(403): verified 5395 rows 2023-03-26 13:55:40,067 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(402): scanned 1827 2023-03-26 13:55:40,067 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(403): verified 5477 rows 2023-03-26 13:55:40,067 INFO [Listener at localhost/41531] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-26 13:55:40,067 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x494529f9 to 127.0.0.1:54332 2023-03-26 13:55:40,067 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:55:40,069 INFO [Listener at localhost/41531] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-26 13:55:40,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-26 13:55:40,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-26 13:55:40,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-26 13:55:40,074 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838940073"}]},"ts":"1679838940073"} 2023-03-26 13:55:40,075 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-26 13:55:40,077 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-26 13:55:40,078 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9871eaedebb91319d17cf11612be1e0, UNASSIGN}] 2023-03-26 13:55:40,079 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9871eaedebb91319d17cf11612be1e0, UNASSIGN 2023-03-26 13:55:40,080 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=a9871eaedebb91319d17cf11612be1e0, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:40,080 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838940080"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838940080"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838940080"}]},"ts":"1679838940080"} 2023-03-26 13:55:40,081 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:55:40,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-26 13:55:40,232 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:40,232 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing a9871eaedebb91319d17cf11612be1e0, disabling compactions & flushes 2023-03-26 13:55:40,232 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:40,232 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:40,232 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. after waiting 0 ms 2023-03-26 13:55:40,232 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:40,233 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing a9871eaedebb91319d17cf11612be1e0 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-26 13:55:40,233 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=A 2023-03-26 13:55:40,233 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:40,233 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=B 2023-03-26 13:55:40,233 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:40,233 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a9871eaedebb91319d17cf11612be1e0, store=C 2023-03-26 13:55:40,233 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:40,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-26 13:55:40,643 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032627c1f438e3264206898724f437cff1ba to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032627c1f438e3264206898724f437cff1ba 2023-03-26 13:55:40,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-26 13:55:41,048 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=434, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/5dfbd051bfe04189b22c64a8cb5c663b 2023-03-26 13:55:41,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-26 13:55:41,456 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/57f9928da8eb453aaff1cd6a894db64a 2023-03-26 13:55:41,863 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=434 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/fa124fb6436c4bb08b65347472f9ca08 2023-03-26 13:55:41,866 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/A/5dfbd051bfe04189b22c64a8cb5c663b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5dfbd051bfe04189b22c64a8cb5c663b 2023-03-26 13:55:41,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5dfbd051bfe04189b22c64a8cb5c663b, entries=100, sequenceid=434, filesize=18.6 K 2023-03-26 13:55:41,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/B/57f9928da8eb453aaff1cd6a894db64a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/57f9928da8eb453aaff1cd6a894db64a 2023-03-26 13:55:41,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/57f9928da8eb453aaff1cd6a894db64a, entries=100, sequenceid=434, filesize=9.5 K 2023-03-26 13:55:41,872 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/.tmp/C/fa124fb6436c4bb08b65347472f9ca08 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/fa124fb6436c4bb08b65347472f9ca08 2023-03-26 13:55:41,875 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/fa124fb6436c4bb08b65347472f9ca08, entries=100, sequenceid=434, filesize=9.5 K 2023-03-26 13:55:41,876 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for a9871eaedebb91319d17cf11612be1e0 in 1643ms, sequenceid=434, compaction requested=true 2023-03-26 13:55:41,882 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/recovered.edits/437.seqid, newMaxSeqId=437, maxSeqId=4 2023-03-26 13:55:41,883 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0. 2023-03-26 13:55:41,883 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for a9871eaedebb91319d17cf11612be1e0: 2023-03-26 13:55:41,884 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:41,884 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=a9871eaedebb91319d17cf11612be1e0, regionState=CLOSED 2023-03-26 13:55:41,885 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838941884"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679838941884"}]},"ts":"1679838941884"} 2023-03-26 13:55:41,886 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-03-26 13:55:41,886 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure a9871eaedebb91319d17cf11612be1e0, server=jenkins-hbase4.apache.org,42517,1679838814721 in 1.8040 sec 2023-03-26 13:55:41,887 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-03-26 13:55:41,887 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a9871eaedebb91319d17cf11612be1e0, UNASSIGN in 1.8080 sec 2023-03-26 13:55:41,888 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838941888"}]},"ts":"1679838941888"} 2023-03-26 13:55:41,888 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-26 13:55:41,889 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-26 13:55:41,891 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8210 sec 2023-03-26 13:55:42,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-26 13:55:42,177 INFO [Listener at localhost/41531] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-03-26 13:55:42,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-26 13:55:42,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:55:42,180 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:55:42,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-26 13:55:42,181 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:55:42,185 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:42,187 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A, FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B, FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C, FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/recovered.edits] 2023-03-26 13:55:42,189 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/2e7883797f5440049e2be9497bfc37ef to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/2e7883797f5440049e2be9497bfc37ef 2023-03-26 13:55:42,190 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5dfbd051bfe04189b22c64a8cb5c663b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/5dfbd051bfe04189b22c64a8cb5c663b 2023-03-26 13:55:42,191 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/ba8e924755da4e9f95934ab4f5f5a93a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/A/ba8e924755da4e9f95934ab4f5f5a93a 2023-03-26 13:55:42,193 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/384d93c3e2a84f7e8c40ed7659685f5c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/384d93c3e2a84f7e8c40ed7659685f5c 2023-03-26 13:55:42,194 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/57f9928da8eb453aaff1cd6a894db64a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/57f9928da8eb453aaff1cd6a894db64a 2023-03-26 13:55:42,195 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/718f66eb291f43e19abbbcc1b1ff70c6 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/B/718f66eb291f43e19abbbcc1b1ff70c6 2023-03-26 13:55:42,197 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/0499e46b49ff40fe8e977c2f66bb6872 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/0499e46b49ff40fe8e977c2f66bb6872 2023-03-26 13:55:42,198 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/5bb19783e2d2432cb8fa15c5a467d98b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/5bb19783e2d2432cb8fa15c5a467d98b 2023-03-26 13:55:42,199 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/fa124fb6436c4bb08b65347472f9ca08 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/C/fa124fb6436c4bb08b65347472f9ca08 2023-03-26 13:55:42,201 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/recovered.edits/437.seqid to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0/recovered.edits/437.seqid 2023-03-26 13:55:42,201 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/a9871eaedebb91319d17cf11612be1e0 2023-03-26 13:55:42,201 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 13:55:42,202 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-26 13:55:42,202 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-26 13:55:42,204 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326022c9e96aae9499fab61eca14fa5071a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326022c9e96aae9499fab61eca14fa5071a 2023-03-26 13:55:42,205 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326091534583a7a4c849b1e425fbdc13778 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326091534583a7a4c849b1e425fbdc13778 2023-03-26 13:55:42,206 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260adfed3fa40a4d54b4416c82054ad710 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260adfed3fa40a4d54b4416c82054ad710 2023-03-26 13:55:42,206 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260b2d93648354458596fe494f8f9f2df8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260b2d93648354458596fe494f8f9f2df8 2023-03-26 13:55:42,207 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303261c1a18e74cce48f086ee32e563c590fd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303261c1a18e74cce48f086ee32e563c590fd 2023-03-26 13:55:42,208 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303261f5fd12b940d4e009eca7d25c470eae0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303261f5fd12b940d4e009eca7d25c470eae0 2023-03-26 13:55:42,209 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032620800782c94a4faa82fd214c216e1b31 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032620800782c94a4faa82fd214c216e1b31 2023-03-26 13:55:42,210 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032623ca1082e8974133a086e1e7b0e67d41 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032623ca1082e8974133a086e1e7b0e67d41 2023-03-26 13:55:42,211 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032624f092d785dc472a8a1173b351da3010 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032624f092d785dc472a8a1173b351da3010 2023-03-26 13:55:42,211 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032626bd286001554ec293475e28fc0742d0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032626bd286001554ec293475e28fc0742d0 2023-03-26 13:55:42,212 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326277574c95f4644118844b324721988a7 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326277574c95f4644118844b324721988a7 2023-03-26 13:55:42,213 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032627c1f438e3264206898724f437cff1ba to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032627c1f438e3264206898724f437cff1ba 2023-03-26 13:55:42,214 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326382594dc29c241d6aad8a6fdda17b2ba to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326382594dc29c241d6aad8a6fdda17b2ba 2023-03-26 13:55:42,215 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263b58116b94444022bc6ef85e0d70208b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263b58116b94444022bc6ef85e0d70208b 2023-03-26 13:55:42,216 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303264036d9c024db4199af987d9a0def7b0e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303264036d9c024db4199af987d9a0def7b0e 2023-03-26 13:55:42,217 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326421467581688423ca459a526632cea5c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326421467581688423ca459a526632cea5c 2023-03-26 13:55:42,217 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032655837411e86e4439810075b0e69f2779 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032655837411e86e4439810075b0e69f2779 2023-03-26 13:55:42,218 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326803f5ea02daf40fba0dd03ef1290f62f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326803f5ea02daf40fba0dd03ef1290f62f 2023-03-26 13:55:42,219 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303268284d7e73075478c9c9b690ef8422f58 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303268284d7e73075478c9c9b690ef8422f58 2023-03-26 13:55:42,220 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326b5b949be63bd4990bbb3c4ff7f14afff to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326b5b949be63bd4990bbb3c4ff7f14afff 2023-03-26 13:55:42,221 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326da5e760513e84648abf523e6552632ba to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326da5e760513e84648abf523e6552632ba 2023-03-26 13:55:42,222 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e71d95465d984d5ebdb6f2012869f3b1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e71d95465d984d5ebdb6f2012869f3b1 2023-03-26 13:55:42,223 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f290fe9766ea40dfa610a214bbdc2cc3 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f290fe9766ea40dfa610a214bbdc2cc3 2023-03-26 13:55:42,223 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-26 13:55:42,225 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:55:42,226 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-26 13:55:42,227 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-26 13:55:42,228 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:55:42,228 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-26 13:55:42,228 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679838942228"}]},"ts":"9223372036854775807"} 2023-03-26 13:55:42,230 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-26 13:55:42,230 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a9871eaedebb91319d17cf11612be1e0, NAME => 'TestAcidGuarantees,,1679838906339.a9871eaedebb91319d17cf11612be1e0.', STARTKEY => '', ENDKEY => ''}] 2023-03-26 13:55:42,230 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-26 13:55:42,230 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679838942230"}]},"ts":"9223372036854775807"} 2023-03-26 13:55:42,231 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-26 13:55:42,232 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:55:42,233 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 54 msec 2023-03-26 13:55:42,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-26 13:55:42,282 INFO [Listener at localhost/41531] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-03-26 13:55:42,294 INFO [Listener at localhost/41531] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 224) - Thread LEAK? -, OpenFileDescriptor=510 (was 529), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=334 (was 401), ProcessCount=171 (was 172), AvailableMemoryMB=8620 (was 8748) 2023-03-26 13:55:42,307 INFO [Listener at localhost/41531] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=510, MaxFileDescriptor=60000, SystemLoadAverage=334, ProcessCount=171, AvailableMemoryMB=8619 2023-03-26 13:55:42,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-26 13:55:42,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 13:55:42,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-26 13:55:42,312 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-26 13:55:42,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-03-26 13:55:42,313 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-26 13:55:42,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-26 13:55:42,314 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:42,314 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2 empty. 2023-03-26 13:55:42,314 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:42,314 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 13:55:42,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-26 13:55:42,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-26 13:55:42,726 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-26 13:55:42,727 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2990228151772e454b24e80da60e1bb2, NAME => 'TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp 2023-03-26 13:55:42,737 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:55:42,737 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2990228151772e454b24e80da60e1bb2, disabling compactions & flushes 2023-03-26 13:55:42,737 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:42,737 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:42,737 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. after waiting 0 ms 2023-03-26 13:55:42,737 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:42,737 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:42,737 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:42,739 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-26 13:55:42,740 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838942740"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679838942740"}]},"ts":"1679838942740"} 2023-03-26 13:55:42,741 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-26 13:55:42,742 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-26 13:55:42,742 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838942742"}]},"ts":"1679838942742"} 2023-03-26 13:55:42,742 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-26 13:55:42,746 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2990228151772e454b24e80da60e1bb2, ASSIGN}] 2023-03-26 13:55:42,748 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2990228151772e454b24e80da60e1bb2, ASSIGN 2023-03-26 13:55:42,748 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2990228151772e454b24e80da60e1bb2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42517,1679838814721; forceNewPlan=false, retain=false 2023-03-26 13:55:42,899 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=2990228151772e454b24e80da60e1bb2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:42,899 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838942899"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838942899"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838942899"}]},"ts":"1679838942899"} 2023-03-26 13:55:42,901 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:55:42,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-26 13:55:43,054 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:43,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2990228151772e454b24e80da60e1bb2, NAME => 'TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.', STARTKEY => '', ENDKEY => ''} 2023-03-26 13:55:43,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:43,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:55:43,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:43,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:43,056 INFO [StoreOpener-2990228151772e454b24e80da60e1bb2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:43,057 DEBUG [StoreOpener-2990228151772e454b24e80da60e1bb2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A 2023-03-26 13:55:43,057 DEBUG [StoreOpener-2990228151772e454b24e80da60e1bb2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A 2023-03-26 13:55:43,057 INFO [StoreOpener-2990228151772e454b24e80da60e1bb2-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:55:43,057 INFO [StoreOpener-2990228151772e454b24e80da60e1bb2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2990228151772e454b24e80da60e1bb2 columnFamilyName A 2023-03-26 13:55:43,058 INFO [StoreOpener-2990228151772e454b24e80da60e1bb2-1] regionserver.HStore(310): Store=2990228151772e454b24e80da60e1bb2/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:55:43,058 INFO [StoreOpener-2990228151772e454b24e80da60e1bb2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:43,059 DEBUG [StoreOpener-2990228151772e454b24e80da60e1bb2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B 2023-03-26 13:55:43,059 DEBUG [StoreOpener-2990228151772e454b24e80da60e1bb2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B 2023-03-26 13:55:43,059 INFO [StoreOpener-2990228151772e454b24e80da60e1bb2-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:55:43,059 INFO [StoreOpener-2990228151772e454b24e80da60e1bb2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2990228151772e454b24e80da60e1bb2 columnFamilyName B 2023-03-26 13:55:43,059 INFO [StoreOpener-2990228151772e454b24e80da60e1bb2-1] regionserver.HStore(310): Store=2990228151772e454b24e80da60e1bb2/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:55:43,060 INFO [StoreOpener-2990228151772e454b24e80da60e1bb2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:43,060 DEBUG [StoreOpener-2990228151772e454b24e80da60e1bb2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C 2023-03-26 13:55:43,061 DEBUG [StoreOpener-2990228151772e454b24e80da60e1bb2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C 2023-03-26 13:55:43,061 INFO [StoreOpener-2990228151772e454b24e80da60e1bb2-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:55:43,061 INFO [StoreOpener-2990228151772e454b24e80da60e1bb2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2990228151772e454b24e80da60e1bb2 columnFamilyName C 2023-03-26 13:55:43,061 INFO [StoreOpener-2990228151772e454b24e80da60e1bb2-1] regionserver.HStore(310): Store=2990228151772e454b24e80da60e1bb2/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:55:43,061 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:43,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:43,062 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:43,063 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 13:55:43,064 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:43,065 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-26 13:55:43,065 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2990228151772e454b24e80da60e1bb2; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=59707642, jitterRate=-0.11028680205345154}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 13:55:43,065 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:43,066 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., pid=49, masterSystemTime=1679838943052 2023-03-26 13:55:43,067 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:43,067 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:43,067 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=2990228151772e454b24e80da60e1bb2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,067 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838943067"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679838943067"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679838943067"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679838943067"}]},"ts":"1679838943067"} 2023-03-26 13:55:43,069 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-03-26 13:55:43,069 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 in 167 msec 2023-03-26 13:55:43,070 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-03-26 13:55:43,070 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2990228151772e454b24e80da60e1bb2, ASSIGN in 323 msec 2023-03-26 13:55:43,071 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-26 13:55:43,071 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838943071"}]},"ts":"1679838943071"} 2023-03-26 13:55:43,072 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-26 13:55:43,074 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-26 13:55:43,074 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 764 msec 2023-03-26 13:55:43,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-26 13:55:43,416 INFO [Listener at localhost/41531] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-03-26 13:55:43,417 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x2c8fbc35 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:43,421 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66369677, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:43,422 DEBUG [Listener at localhost/41531] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:43,424 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59270, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:43,425 DEBUG [Listener at localhost/41531] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-26 13:55:43,426 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43140, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-26 13:55:43,427 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x3fb41c50 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:43,431 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e0b6187, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:43,431 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x357cb237 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:43,436 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ab56e54, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:43,437 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x4539ed0c to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:43,440 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15840296, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:43,441 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x32cd672d to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:43,444 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3816bdf5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:43,444 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x09b8ff30 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:43,448 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@50150525, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:43,448 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x028a3a77 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:43,455 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a4e3b2e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:43,455 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x4cefb012 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:43,459 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f65627a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:43,459 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x24a917df to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:43,466 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15f74c78, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:43,466 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x78fa3dee to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:43,470 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1401cac1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:43,471 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x36d2d826 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:55:43,478 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f6e865f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:55:43,482 DEBUG [hconnection-0x6ed49704-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:43,482 DEBUG [hconnection-0xa6da1e2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:43,482 DEBUG [hconnection-0x7f184222-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:43,484 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59284, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:43,484 DEBUG [hconnection-0x35da950d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:43,484 DEBUG [hconnection-0x4a1eb9bd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:43,488 DEBUG [hconnection-0x6dd440ec-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:43,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:43,492 DEBUG [hconnection-0x60f0c745-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:43,492 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59286, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:43,492 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59294, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:43,492 DEBUG [hconnection-0x1108d0cc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:43,492 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59310, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:43,492 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59314, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:43,492 DEBUG [hconnection-0x5d6edb70-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:43,492 DEBUG [hconnection-0x11dc770b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:55:43,498 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59320, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:43,499 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59336, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:43,500 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59326, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:43,500 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59340, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:43,503 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59354, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:55:43,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:43,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:43,503 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:43,503 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:43,503 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:43,503 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:43,504 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,504 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,508 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,508 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:43,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:43,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:43,509 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,509 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:43,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:43,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:43,514 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:43,515 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 13:55:43,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:43,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:43,516 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:43,516 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:43,516 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:43,516 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:43,516 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:43,516 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:43,516 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,516 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:43,516 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:43,517 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:43,517 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:43,517 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:43,518 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,518 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:43,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:43,518 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,518 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:43,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:43,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:43,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f5de31[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:43,520 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:43,520 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,521 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,521 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:43,521 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:43,521 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:43,523 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,523 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:43,523 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:43,523 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:43,524 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:43,524 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:43,524 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:43,524 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:43,524 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:43,524 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:43,524 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:43,526 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,526 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:43,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,526 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:43,526 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:43,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:43,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:43,527 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:43,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,529 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,529 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,529 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,529 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,530 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,530 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:43,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:43,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:43,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:43,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:43,530 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:43,530 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,530 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,530 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:43,530 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:43,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:43,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:43,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:43,530 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:43,532 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/155b936dc9c54663b8d709c2e00a1629 2023-03-26 13:55:43,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:43,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839003534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:43,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839003535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,540 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:43,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839003538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:43,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839003539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:43,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839003540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,630 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:43,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:43,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:43,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:43,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:43,635 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:43,635 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:43,635 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:43,635 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:43,635 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,635 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,637 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,637 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:43,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:43,637 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,637 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:43,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:43,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:43,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:43,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:43,640 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:43,641 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:43,641 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:43,641 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:43,641 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:43,641 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:43,641 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:43,641 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:43,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839003640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,641 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:43,641 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:43,641 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:43,641 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:43,641 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,641 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:43,641 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:43,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:43,644 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,644 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:43,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:43,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839003640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,644 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,644 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:43,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:43,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839003641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:43,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:43,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:43,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839003643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839003643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:43,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:43,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:43,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55a68799[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:43,646 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:43,646 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,647 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,647 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:43,647 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:43,647 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:43,647 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,647 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:43,649 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,649 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:43,649 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:43,649 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:43,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:43,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:43,651 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:43,651 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:43,651 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:43,651 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:43,651 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:43,652 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:43,652 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,653 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:43,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:43,653 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:43,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,654 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:43,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,657 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:43,657 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:43,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:43,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:43,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:43,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:43,657 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:43,657 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,657 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,658 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:43,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:43,658 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:43,658 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:43,658 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,658 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,658 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,658 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,758 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:43,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:43,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:43,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:43,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:43,761 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:43,761 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:43,761 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:43,761 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:43,761 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,761 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,763 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:43,763 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:43,763 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,763 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:43,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:43,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:43,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:43,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:43,764 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:43,764 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:43,764 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:43,764 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:43,764 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:43,764 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:43,764 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:43,765 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:43,765 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:43,765 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:43,765 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:43,765 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,765 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:43,765 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:43,766 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,766 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:43,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:43,767 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,767 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:43,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:43,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a4878a2[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:43,768 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:43,768 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,769 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,769 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:43,769 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:43,769 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:43,770 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,770 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:43,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,771 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:43,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:43,771 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:43,771 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:43,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:43,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:43,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:43,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:43,773 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:43,774 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:43,774 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:43,774 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:43,774 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:43,776 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,776 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:43,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:43,776 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:43,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:43,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:43,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:43,777 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:43,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,778 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:43,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:43,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:43,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:43,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:43,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:43,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:43,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,780 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:43,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:43,780 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:43,780 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,781 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:43,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:43,781 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,781 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,846 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:43,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839003843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:43,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839003845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:43,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839003846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:43,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839003846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:43,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839003847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,881 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:43,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:43,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:43,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:43,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:43,884 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:43,884 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:43,884 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:43,884 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:43,884 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,884 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,885 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,885 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:43,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:43,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:43,885 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,885 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:43,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:43,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:43,886 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:43,886 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:43,886 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:43,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:43,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:43,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:43,887 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:43,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:43,887 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:43,887 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:43,887 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:43,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:43,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:43,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:43,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:43,893 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:43,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:43,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2becd8d6[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:43,894 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:43,894 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,895 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,898 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,898 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,898 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:43,898 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:43,898 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:43,898 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:43,899 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,899 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:43,900 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:43,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:43,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:43,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:43,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:43,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:43,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:43,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:43,904 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:43,904 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:43,904 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:43,904 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:43,904 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:43,907 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,907 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:43,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,907 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:43,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:43,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,908 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:43,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,910 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:43,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:43,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:43,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:43,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:43,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:43,913 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:43,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:43,913 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:43,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:43,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:43,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:43,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:43,913 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:43,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:43,914 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:43,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:43,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:43,945 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/42ba44fa7021473fb680205385fe5bdf 2023-03-26 13:55:44,013 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:44,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:44,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:44,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:44,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:44,016 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:44,016 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:44,016 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:44,016 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:44,017 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,017 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,018 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:44,018 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:44,018 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,018 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:44,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:44,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:44,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:44,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:44,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:44,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:44,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:44,020 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:44,020 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:44,020 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:44,020 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:44,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:44,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:44,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:44,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:44,022 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,022 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:44,022 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,022 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:44,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:44,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:44,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:44,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c78fcc[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:44,023 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:44,023 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,025 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,025 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:44,025 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:44,025 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:44,025 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,025 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:44,026 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:44,027 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:44,026 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,027 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:44,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:44,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:44,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:44,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:44,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:44,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:44,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:44,029 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:44,029 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:44,029 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:44,029 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:44,029 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:44,030 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,030 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:44,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:44,030 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:44,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:44,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:44,031 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:44,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:44,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,032 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:44,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,034 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,034 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,034 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:44,034 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:44,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:44,034 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:44,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:44,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:44,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:44,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:44,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:44,035 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:44,034 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,035 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:44,035 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,035 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,035 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,035 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,135 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:44,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:44,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:44,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:44,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:44,138 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:44,138 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:44,138 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:44,138 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:44,138 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,138 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,141 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:44,141 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:44,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:44,141 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,141 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:44,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:44,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:44,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:44,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:44,142 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:44,142 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:44,142 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:44,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:44,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:44,143 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:44,143 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:44,143 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:44,143 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:44,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:44,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:44,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:44,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:44,144 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,144 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:44,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:44,146 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,146 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:44,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:44,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:44,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@653c5a82[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:44,148 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:44,148 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,149 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,149 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,149 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:44,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,149 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:44,149 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:44,149 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:44,151 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:44,151 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:44,151 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:44,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:44,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839004149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:44,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:44,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839004152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:44,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839004152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:44,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:44,154 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:44,154 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:44,155 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:44,155 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:44,155 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:44,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:44,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839004152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:44,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839004153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,157 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,157 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:44,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:44,157 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:44,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:44,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,158 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,159 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:44,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,161 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:44,161 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:44,161 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:44,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:44,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:44,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:44,161 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,161 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:44,161 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,162 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,162 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:44,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:44,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:44,162 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,162 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:44,162 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,162 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,262 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:44,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:44,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:44,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:44,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:44,265 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:44,265 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:44,265 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:44,265 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:44,265 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,265 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,266 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,266 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:44,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:44,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:44,267 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,267 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:44,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:44,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:44,268 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:44,268 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:44,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:44,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:44,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:44,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:44,268 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:44,268 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:44,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:44,268 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:44,268 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:44,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:44,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:44,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:44,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:44,270 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,270 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:44,270 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:44,270 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:44,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:44,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:44,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cf8b3ef[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:44,271 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:44,271 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,273 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,274 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:44,274 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:44,274 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,274 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:44,274 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:44,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,276 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:44,276 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:44,276 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:44,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:44,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:44,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:44,278 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:44,278 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:44,278 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:44,278 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:44,278 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:44,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:44,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:44,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:44,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:44,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,282 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,284 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:44,284 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:44,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:44,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:44,284 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,284 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,284 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,284 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,284 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,284 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,284 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,284 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:44,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:44,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:44,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:44,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:44,285 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:44,285 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:44,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/6b03de4565424ceca121924c02578f81 2023-03-26 13:55:44,357 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/155b936dc9c54663b8d709c2e00a1629 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/155b936dc9c54663b8d709c2e00a1629 2023-03-26 13:55:44,361 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/155b936dc9c54663b8d709c2e00a1629, entries=100, sequenceid=15, filesize=9.3 K 2023-03-26 13:55:44,361 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/42ba44fa7021473fb680205385fe5bdf as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/42ba44fa7021473fb680205385fe5bdf 2023-03-26 13:55:44,364 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/42ba44fa7021473fb680205385fe5bdf, entries=100, sequenceid=15, filesize=9.3 K 2023-03-26 13:55:44,365 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/6b03de4565424ceca121924c02578f81 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/6b03de4565424ceca121924c02578f81 2023-03-26 13:55:44,368 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/6b03de4565424ceca121924c02578f81, entries=100, sequenceid=15, filesize=9.3 K 2023-03-26 13:55:44,369 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 2990228151772e454b24e80da60e1bb2 in 854ms, sequenceid=15, compaction requested=false 2023-03-26 13:55:44,369 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-26 13:55:44,369 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:44,385 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:44,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:44,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:44,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:44,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:44,388 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:44,388 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:44,388 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:44,388 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:44,388 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,388 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,396 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,396 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:44,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:44,396 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,396 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:44,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:44,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:44,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:44,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:44,397 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:44,397 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:44,397 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:44,398 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:44,398 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:44,398 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:44,398 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:44,398 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-26 13:55:44,398 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:44,398 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:44,399 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:44,399 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:44,399 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:44,399 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:44,410 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/0f5457e1beca41bcab9d6580cf2fec2f 2023-03-26 13:55:44,444 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/bca19ce9987a4300977b8c34d9a96fda 2023-03-26 13:55:44,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:44,658 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:44,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:44,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839004664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:44,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839004665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:44,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839004665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:44,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839004666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:44,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839004670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:44,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:44,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839004771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839004771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:44,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839004773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:44,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839004773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:44,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839004777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,866 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/3cd68ae5d8254b20975a41788f0eb484 2023-03-26 13:55:44,869 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/0f5457e1beca41bcab9d6580cf2fec2f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/0f5457e1beca41bcab9d6580cf2fec2f 2023-03-26 13:55:44,872 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/0f5457e1beca41bcab9d6580cf2fec2f, entries=150, sequenceid=34, filesize=11.6 K 2023-03-26 13:55:44,873 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/bca19ce9987a4300977b8c34d9a96fda as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/bca19ce9987a4300977b8c34d9a96fda 2023-03-26 13:55:44,876 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/bca19ce9987a4300977b8c34d9a96fda, entries=150, sequenceid=34, filesize=11.6 K 2023-03-26 13:55:44,877 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/3cd68ae5d8254b20975a41788f0eb484 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/3cd68ae5d8254b20975a41788f0eb484 2023-03-26 13:55:44,880 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/3cd68ae5d8254b20975a41788f0eb484, entries=150, sequenceid=34, filesize=11.6 K 2023-03-26 13:55:44,881 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 2990228151772e454b24e80da60e1bb2 in 483ms, sequenceid=34, compaction requested=false 2023-03-26 13:55:44,882 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:44,882 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:44,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:44,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:44,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:44,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:44,884 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:44,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:44,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:44,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:44,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@619ff4ce[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:44,886 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:44,886 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,887 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:44,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:44,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:44,887 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,887 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:44,889 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:44,889 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:44,889 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:44,889 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:44,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:44,891 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:44,891 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:44,891 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:44,891 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:44,892 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:44,894 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,894 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:44,894 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:44,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:44,895 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:44,896 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,898 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,898 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,899 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:44,899 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:44,899 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:44,899 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:44,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:44,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:44,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:44,899 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:44,899 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:44,899 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,899 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:44,899 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:44,899 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:44,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:44,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:44,980 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-26 13:55:44,981 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:44,981 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:44,981 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:44,981 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:44,981 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:44,981 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:44,988 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/e5114cfa21574166b7a9403bb617f46f 2023-03-26 13:55:44,999 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:45,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:45,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839004993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:45,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:45,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:45,002 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:45,002 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:45,002 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:45,002 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:45,002 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,002 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,004 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:45,004 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:45,004 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,004 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:45,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:45,005 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:45,005 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:45,005 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:45,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839004997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839004998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,006 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839004999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839005000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:45,011 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:45,011 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:45,011 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:45,011 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:45,011 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:45,013 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,013 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,013 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,013 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:45,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@591eaede[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:45,014 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:45,014 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,019 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:45,019 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:45,019 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,019 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:45,020 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,021 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,021 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:45,021 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:45,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:45,023 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:45,023 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:45,023 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:45,023 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:45,023 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:45,025 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,025 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,025 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:45,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,025 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,026 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:45,029 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:45,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:45,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:45,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:45,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:45,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,029 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:45,029 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:45,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:45,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:45,102 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839005101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839005107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839005107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839005107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839005108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,129 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:45,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:45,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:45,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:45,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:45,132 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:45,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:45,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:45,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:45,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,135 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,135 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:45,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:45,135 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,135 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:45,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:45,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:45,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:45,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:45,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:45,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:45,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:45,137 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,137 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:45,137 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:45,137 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:45,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:45,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:45,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:45,139 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,139 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,140 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,140 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:45,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e710735[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:45,141 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:45,141 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,142 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,142 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:45,142 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:45,142 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:45,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,143 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:45,144 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,144 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,144 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:45,144 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:45,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:45,146 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:45,146 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:45,146 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:45,146 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:45,146 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:45,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:45,153 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,154 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:45,158 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:45,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:45,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:45,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:45,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:45,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:45,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:45,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:45,158 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:45,159 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:45,159 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:45,259 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:45,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:45,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:45,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:45,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:45,262 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:45,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:45,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:45,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:45,263 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,263 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,264 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:45,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:45,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:45,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:45,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:45,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:45,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:45,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:45,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:45,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:45,265 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,266 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:45,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:45,266 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:45,266 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:45,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:45,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:45,266 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:45,267 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,267 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,267 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:45,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37ffb0df[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:45,269 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:45,269 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,271 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,271 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:45,272 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,272 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:45,272 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:45,272 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:45,273 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:45,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:45,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:45,275 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:45,275 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:45,275 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:45,275 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:45,275 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:45,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:45,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,277 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,278 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:45,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:45,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:45,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:45,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:45,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,280 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:45,280 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:45,280 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:45,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:45,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:45,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:45,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839005303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839005309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839005310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839005310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,314 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839005312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,380 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:45,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:45,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:45,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:45,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:45,383 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:45,383 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:45,383 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:45,383 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:45,384 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,384 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,386 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:45,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:45,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:45,386 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,386 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,386 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:45,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:45,387 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:45,387 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:45,387 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:45,387 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:45,387 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:45,387 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:45,387 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:45,387 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,387 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:45,388 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:45,388 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:45,388 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:45,389 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,389 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,389 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,389 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:45,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29580b6a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:45,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:45,391 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,391 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,392 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:45,392 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:45,392 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:45,392 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,392 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:45,393 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,393 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:45,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:45,395 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:45,395 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:45,395 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:45,395 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:45,395 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:45,396 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/4deca1641fa44b5c9fc9fdb8f4ecf00e 2023-03-26 13:55:45,398 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,398 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,398 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:45,399 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,403 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:45,404 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:45,404 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:45,404 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:45,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:45,404 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:45,404 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:45,404 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:45,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:45,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:45,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:45,404 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,405 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,405 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,405 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,405 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,405 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/c2531883b9434d058a61d0fe91716054 2023-03-26 13:55:45,408 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/e5114cfa21574166b7a9403bb617f46f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/e5114cfa21574166b7a9403bb617f46f 2023-03-26 13:55:45,411 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/e5114cfa21574166b7a9403bb617f46f, entries=200, sequenceid=50, filesize=13.9 K 2023-03-26 13:55:45,411 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/4deca1641fa44b5c9fc9fdb8f4ecf00e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/4deca1641fa44b5c9fc9fdb8f4ecf00e 2023-03-26 13:55:45,414 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/4deca1641fa44b5c9fc9fdb8f4ecf00e, entries=150, sequenceid=50, filesize=11.6 K 2023-03-26 13:55:45,414 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/c2531883b9434d058a61d0fe91716054 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c2531883b9434d058a61d0fe91716054 2023-03-26 13:55:45,418 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c2531883b9434d058a61d0fe91716054, entries=150, sequenceid=50, filesize=11.6 K 2023-03-26 13:55:45,418 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 2990228151772e454b24e80da60e1bb2 in 438ms, sequenceid=50, compaction requested=true 2023-03-26 13:55:45,419 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:45,419 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:45,419 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:45,419 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:45,419 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:45,419 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:45,420 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35654 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:45,420 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/A is initiating minor compaction (all files) 2023-03-26 13:55:45,420 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/A in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,420 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/155b936dc9c54663b8d709c2e00a1629, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/0f5457e1beca41bcab9d6580cf2fec2f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/e5114cfa21574166b7a9403bb617f46f] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=34.8 K 2023-03-26 13:55:45,420 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33314 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:45,420 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/B is initiating minor compaction (all files) 2023-03-26 13:55:45,421 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/B in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,421 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/42ba44fa7021473fb680205385fe5bdf, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/bca19ce9987a4300977b8c34d9a96fda, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/4deca1641fa44b5c9fc9fdb8f4ecf00e] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=32.5 K 2023-03-26 13:55:45,421 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 155b936dc9c54663b8d709c2e00a1629, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679838943515 2023-03-26 13:55:45,421 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 42ba44fa7021473fb680205385fe5bdf, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679838943515 2023-03-26 13:55:45,421 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 0f5457e1beca41bcab9d6580cf2fec2f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1679838943533 2023-03-26 13:55:45,422 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting bca19ce9987a4300977b8c34d9a96fda, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1679838943533 2023-03-26 13:55:45,422 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting e5114cfa21574166b7a9403bb617f46f, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679838944663 2023-03-26 13:55:45,422 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 4deca1641fa44b5c9fc9fdb8f4ecf00e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679838944663 2023-03-26 13:55:45,429 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#A#compaction#447 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:45,429 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#B#compaction#448 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:45,504 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:45,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:45,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:45,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:45,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:45,507 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:45,507 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:45,507 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:45,507 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:45,508 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,508 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,511 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,511 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:45,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:45,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,512 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,512 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:45,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:45,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:45,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:45,513 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:45,513 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:45,513 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:45,513 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:45,513 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:45,513 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:45,513 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,513 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:45,513 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-26 13:55:45,514 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:45,514 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:45,514 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:45,514 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:45,514 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:45,514 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:45,522 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/7e698084699a4dee914a9b48e83a5cb0 2023-03-26 13:55:45,530 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/12525b1709b34a3b874ceadece43e338 2023-03-26 13:55:45,541 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/b3e66f98a3e84b318e66f7f8e0aecdc0 2023-03-26 13:55:45,545 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/7e698084699a4dee914a9b48e83a5cb0 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/7e698084699a4dee914a9b48e83a5cb0 2023-03-26 13:55:45,548 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/7e698084699a4dee914a9b48e83a5cb0, entries=150, sequenceid=66, filesize=11.6 K 2023-03-26 13:55:45,548 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/12525b1709b34a3b874ceadece43e338 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/12525b1709b34a3b874ceadece43e338 2023-03-26 13:55:45,551 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/12525b1709b34a3b874ceadece43e338, entries=150, sequenceid=66, filesize=11.6 K 2023-03-26 13:55:45,552 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/b3e66f98a3e84b318e66f7f8e0aecdc0 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b3e66f98a3e84b318e66f7f8e0aecdc0 2023-03-26 13:55:45,555 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b3e66f98a3e84b318e66f7f8e0aecdc0, entries=150, sequenceid=66, filesize=11.6 K 2023-03-26 13:55:45,556 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 2990228151772e454b24e80da60e1bb2 in 43ms, sequenceid=66, compaction requested=true 2023-03-26 13:55:45,556 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:45,556 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,556 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:45,556 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:45,556 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,556 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:45,556 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:45,557 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,557 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,558 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,558 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:45,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59c28c1[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:45,560 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:45,560 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,562 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,562 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:45,562 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:45,562 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:45,562 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,562 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:45,563 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:45,563 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:45,563 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,563 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:45,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:45,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:45,566 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:45,566 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:45,566 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:45,566 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:45,566 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:45,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:45,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,569 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,570 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:45,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:45,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:45,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:45,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:45,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:45,580 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:45,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:45,581 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:45,581 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:45,581 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,581 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,581 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,581 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,581 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,581 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,581 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:45,619 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:55:45,619 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:45,619 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:45,619 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:45,619 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:45,619 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:45,619 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:45,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679838945627, max timestamp=1679838945650Num uniques 859; 2023-03-26 13:55:45,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679838945627, max timestamp=1679838945650Num uniques 859; 2023-03-26 13:55:45,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-26 13:55:45,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679838945627, max timestamp=1679838945650Num uniques 859; 2023-03-26 13:55:45,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-26 13:55:45,652 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-26 13:55:45,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839005669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839005676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,681 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:45,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:45,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:45,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:45,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:45,684 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:45,684 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:45,684 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:45,684 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:45,684 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,685 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,686 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:45,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:45,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:45,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:45,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:45,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:45,687 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:45,687 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:45,687 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:45,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,691 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:45,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839005678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,692 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,692 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:45,692 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,692 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:45,692 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,692 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839005678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839005678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:45,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:45,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:45,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:45,695 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:45,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5382cfe8[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:45,697 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:45,697 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,698 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,698 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:45,698 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:45,698 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:45,698 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,698 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:45,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,701 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:45,701 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,701 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:45,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:45,706 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:45,706 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:45,706 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:45,706 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:45,706 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:45,707 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,707 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,707 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:45,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,708 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:45,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,710 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,712 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,712 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,712 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:45,712 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:45,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:45,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:45,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:45,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:45,712 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:45,713 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:45,712 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,713 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:45,713 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,713 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,713 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,713 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:45,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839005779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839005779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839005793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839005793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839005793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,813 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:45,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:45,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:45,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:45,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:45,816 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:45,816 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:45,816 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:45,816 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:45,816 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,816 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,818 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,818 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:45,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:45,818 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,818 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:45,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:45,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,819 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:45,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:45,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:45,819 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:45,819 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:45,819 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:45,819 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:45,819 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:45,819 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,819 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:45,820 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:45,820 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,820 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:45,820 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:45,820 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,820 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:45,820 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:45,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,822 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,823 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,823 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:45,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1931f40f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:45,824 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:45,824 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,825 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:45,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:45,825 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,825 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:45,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:45,827 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,827 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,827 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:45,827 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:45,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:45,829 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:45,829 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:45,829 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:45,829 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:45,829 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:45,830 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,830 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,830 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:45,831 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,832 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,835 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,835 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:45,835 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,835 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,835 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:45,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:45,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:45,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:45,836 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:45,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:45,836 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,836 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,836 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,836 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:45,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:45,836 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:45,836 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:45,837 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/4446a5c2924b44b3a61a50facf090991 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/4446a5c2924b44b3a61a50facf090991 2023-03-26 13:55:45,841 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/A of 2990228151772e454b24e80da60e1bb2 into 4446a5c2924b44b3a61a50facf090991(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:45,841 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:45,841 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/A, priority=13, startTime=1679838945419; duration=0sec 2023-03-26 13:55:45,842 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:45,842 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:55:45,843 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:55:45,843 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/C is initiating minor compaction (all files) 2023-03-26 13:55:45,843 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/C in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,843 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/6b03de4565424ceca121924c02578f81, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/3cd68ae5d8254b20975a41788f0eb484, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c2531883b9434d058a61d0fe91716054, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b3e66f98a3e84b318e66f7f8e0aecdc0] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=44.1 K 2023-03-26 13:55:45,843 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 6b03de4565424ceca121924c02578f81, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679838943515 2023-03-26 13:55:45,844 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 3cd68ae5d8254b20975a41788f0eb484, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1679838943533 2023-03-26 13:55:45,844 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting c2531883b9434d058a61d0fe91716054, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679838944663 2023-03-26 13:55:45,844 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting b3e66f98a3e84b318e66f7f8e0aecdc0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1679838944992 2023-03-26 13:55:45,846 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/e0317c78fe0e416ea1f7399746d978ce as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/e0317c78fe0e416ea1f7399746d978ce 2023-03-26 13:55:45,851 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/B of 2990228151772e454b24e80da60e1bb2 into e0317c78fe0e416ea1f7399746d978ce(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:45,851 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:45,851 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/B, priority=13, startTime=1679838945419; duration=0sec 2023-03-26 13:55:45,851 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:45,852 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#C#compaction#453 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:45,860 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/c0e3e1b29bf14b999d52f01f0e0529bf as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c0e3e1b29bf14b999d52f01f0e0529bf 2023-03-26 13:55:45,867 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2990228151772e454b24e80da60e1bb2/C of 2990228151772e454b24e80da60e1bb2 into c0e3e1b29bf14b999d52f01f0e0529bf(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:45,868 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:45,868 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/C, priority=12, startTime=1679838945419; duration=0sec 2023-03-26 13:55:45,868 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:45,937 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:45,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:45,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:45,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:45,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:45,939 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:45,939 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:45,939 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:45,939 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:45,940 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,940 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,941 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:45,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:45,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:45,941 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,942 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,942 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:45,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:45,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:45,943 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:45,943 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:45,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:45,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:45,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:45,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:45,943 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,943 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:45,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:45,943 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:45,943 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:45,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:45,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:45,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:45,943 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:45,945 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,945 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,945 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:45,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:45,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@705bbf18[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:45,946 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:45,946 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,947 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:45,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:45,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:45,949 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,949 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:45,953 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,953 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:45,954 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,954 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:45,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:45,957 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:45,957 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:45,957 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:45,957 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:45,957 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:45,958 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,958 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,958 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:45,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:45,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,959 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:45,959 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:45,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:45,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:45,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:45,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:45,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:45,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:45,964 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:45,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:45,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,964 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:45,964 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:45,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:45,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:45,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,964 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,965 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:45,965 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:45,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:45,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:45,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839005983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:45,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:45,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839005984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:46,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839005999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:46,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839005999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:46,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839006000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,026 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=76 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/ac47ae6c228042dbb042a684ec91d555 2023-03-26 13:55:46,064 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:46,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:46,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:46,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:46,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:46,067 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:46,067 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:46,067 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:46,067 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:46,068 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,068 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,070 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,070 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:46,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:46,070 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,070 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:46,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:46,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,071 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:46,071 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:46,071 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:46,071 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:46,071 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:46,071 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:46,072 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:46,072 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:46,072 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:46,072 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:46,072 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:46,072 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:46,072 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:46,072 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:46,072 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,072 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:46,072 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:46,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:46,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:46,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14655bf2[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:46,076 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:46,076 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:46,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:46,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:46,079 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,079 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:46,081 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,081 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:46,081 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,081 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:46,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:46,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:46,084 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:46,084 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:46,084 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:46,084 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:46,084 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:46,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:46,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:46,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:46,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,090 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,091 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,094 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,094 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,094 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,094 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:46,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:46,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:46,094 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:46,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:46,095 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:46,095 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:46,094 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:46,094 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:46,095 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,095 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,095 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:46,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:46,195 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:46,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:46,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:46,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:46,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:46,198 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:46,198 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:46,198 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:46,199 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:46,199 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,199 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,200 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,200 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:46,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:46,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:46,200 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,200 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:46,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:46,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:46,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:46,201 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:46,202 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:46,202 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:46,202 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:46,202 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:46,202 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:46,202 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:46,202 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:46,202 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:46,202 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:46,202 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:46,202 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,202 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:46,202 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:46,203 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,203 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,204 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,204 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:46,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:46,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51dd2b19[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:46,205 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:46,205 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,207 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,207 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:46,207 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:46,207 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:46,207 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,207 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:46,208 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:46,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,208 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,208 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:46,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:46,211 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:46,211 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:46,211 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:46,211 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:46,211 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:46,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:46,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:46,213 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,216 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,218 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,218 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,218 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,218 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:46,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:46,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:46,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:46,218 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:46,218 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:46,218 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:46,218 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:46,218 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:46,218 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,218 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:46,218 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:46,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:46,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:46,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839006293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,293 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:46,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839006293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:46,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839006304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:46,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839006304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:46,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839006304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,318 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:46,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:46,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:46,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:46,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:46,322 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:46,322 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:46,322 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:46,322 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:46,322 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,322 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,324 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,324 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:46,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:46,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:46,324 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,324 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:46,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:46,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,326 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:46,326 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:46,326 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:46,326 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:46,326 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:46,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:46,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:46,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:46,327 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:46,327 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:46,327 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:46,327 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:46,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:46,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:46,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:46,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:46,329 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,329 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,329 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,329 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:46,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@475a3ca1[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:46,330 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:46,330 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,333 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,333 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:46,333 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:46,333 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:46,333 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,333 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:46,334 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,334 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:46,334 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:46,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,334 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:46,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:46,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:46,336 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:46,336 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:46,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:46,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:46,337 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:46,339 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,339 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:46,339 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:46,340 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,341 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,343 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:46,343 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,343 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,343 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:46,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:46,343 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:46,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:46,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:46,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:46,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:46,343 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:46,343 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:46,343 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:46,343 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,343 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:46,343 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,343 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,343 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,435 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=76 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/d0a6dac1b87e4811a826f14db0f5dea7 2023-03-26 13:55:46,443 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:46,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:46,445 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=76 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/2d5639a9a1054f04a9fded3472d1b7ce 2023-03-26 13:55:46,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:46,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:46,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:46,447 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:46,447 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:46,447 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:46,447 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:46,447 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,447 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,448 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:46,448 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:46,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:46,448 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/ac47ae6c228042dbb042a684ec91d555 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/ac47ae6c228042dbb042a684ec91d555 2023-03-26 13:55:46,449 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,449 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:46,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:46,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:46,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:46,450 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:46,450 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:46,450 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:46,450 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:46,450 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:46,450 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:46,450 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:46,450 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:46,451 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:46,451 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:46,451 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:46,451 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,451 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:46,451 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:46,452 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/ac47ae6c228042dbb042a684ec91d555, entries=300, sequenceid=76, filesize=18.5 K 2023-03-26 13:55:46,453 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/d0a6dac1b87e4811a826f14db0f5dea7 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/d0a6dac1b87e4811a826f14db0f5dea7 2023-03-26 13:55:46,456 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/d0a6dac1b87e4811a826f14db0f5dea7, entries=150, sequenceid=76, filesize=11.6 K 2023-03-26 13:55:46,456 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/2d5639a9a1054f04a9fded3472d1b7ce as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/2d5639a9a1054f04a9fded3472d1b7ce 2023-03-26 13:55:46,457 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,457 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,457 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,457 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:46,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ad4e82d[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:46,459 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:46,459 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,460 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/2d5639a9a1054f04a9fded3472d1b7ce, entries=150, sequenceid=76, filesize=11.6 K 2023-03-26 13:55:46,460 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,460 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=147.60 KB/151140 for 2990228151772e454b24e80da60e1bb2 in 841ms, sequenceid=76, compaction requested=true 2023-03-26 13:55:46,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,460 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:46,460 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,460 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:46,461 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:46,461 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:46,461 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:46,460 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:46,461 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:46,461 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:46,461 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:46,461 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:46,461 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42781 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:46,462 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/A is initiating minor compaction (all files) 2023-03-26 13:55:46,462 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:46,462 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/A in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:46,462 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/B is initiating minor compaction (all files) 2023-03-26 13:55:46,462 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/4446a5c2924b44b3a61a50facf090991, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/7e698084699a4dee914a9b48e83a5cb0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/ac47ae6c228042dbb042a684ec91d555] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=41.8 K 2023-03-26 13:55:46,462 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/B in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:46,462 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/e0317c78fe0e416ea1f7399746d978ce, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/12525b1709b34a3b874ceadece43e338, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/d0a6dac1b87e4811a826f14db0f5dea7] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=34.9 K 2023-03-26 13:55:46,462 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 4446a5c2924b44b3a61a50facf090991, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679838944663 2023-03-26 13:55:46,462 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,462 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e0317c78fe0e416ea1f7399746d978ce, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679838944663 2023-03-26 13:55:46,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,462 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 7e698084699a4dee914a9b48e83a5cb0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1679838944992 2023-03-26 13:55:46,463 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 12525b1709b34a3b874ceadece43e338, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1679838944992 2023-03-26 13:55:46,462 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:46,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,463 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:46,463 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:46,463 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting ac47ae6c228042dbb042a684ec91d555, keycount=300, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1679838945611 2023-03-26 13:55:46,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,463 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting d0a6dac1b87e4811a826f14db0f5dea7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1679838945611 2023-03-26 13:55:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:46,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:46,465 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:46,466 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:46,466 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:46,466 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:46,466 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:46,468 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,468 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:46,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,468 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:46,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:46,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,469 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,470 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,471 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#A#compaction#456 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:46,471 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#B#compaction#457 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:46,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,474 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:46,474 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:46,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:46,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:46,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:46,475 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,475 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,475 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,475 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:46,475 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:46,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:46,475 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,476 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,476 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:46,476 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:46,476 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:46,486 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/d8bdfb87dcde472c818669b1029cd136 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/d8bdfb87dcde472c818669b1029cd136 2023-03-26 13:55:46,488 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/dfcef5bee5344e3793df5f5f741ba2fe as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/dfcef5bee5344e3793df5f5f741ba2fe 2023-03-26 13:55:46,491 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/B of 2990228151772e454b24e80da60e1bb2 into d8bdfb87dcde472c818669b1029cd136(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:46,491 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:46,491 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/B, priority=13, startTime=1679838946461; duration=0sec 2023-03-26 13:55:46,491 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:46,492 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-26 13:55:46,492 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 13:55:46,492 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 13:55:46,492 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. because compaction request was cancelled 2023-03-26 13:55:46,493 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/A of 2990228151772e454b24e80da60e1bb2 into dfcef5bee5344e3793df5f5f741ba2fe(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:46,493 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:46,493 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/A, priority=13, startTime=1679838946460; duration=0sec 2023-03-26 13:55:46,493 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:46,576 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:46,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:46,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:46,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:46,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:46,579 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:46,579 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:46,579 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:46,579 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:46,580 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,580 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,581 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:46,581 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:46,581 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,581 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:46,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:46,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:46,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:46,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:46,582 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:46,582 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:46,582 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:46,583 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:46,583 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:46,583 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:46,583 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:46,583 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=147.60 KB heapSize=406.08 KB 2023-03-26 13:55:46,583 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:46,583 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:46,583 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:46,583 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:46,583 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:46,583 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:46,590 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/d199f1349acc4b7abf4b0e135df92383 2023-03-26 13:55:46,599 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/10b9c1d3fd864c678aa79ee3d794131b 2023-03-26 13:55:46,608 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=116 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/a9ab7df802b24ec0b4058ed9cb3ebd59 2023-03-26 13:55:46,615 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/d199f1349acc4b7abf4b0e135df92383 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/d199f1349acc4b7abf4b0e135df92383 2023-03-26 13:55:46,618 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/d199f1349acc4b7abf4b0e135df92383, entries=150, sequenceid=116, filesize=11.6 K 2023-03-26 13:55:46,619 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/10b9c1d3fd864c678aa79ee3d794131b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/10b9c1d3fd864c678aa79ee3d794131b 2023-03-26 13:55:46,622 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/10b9c1d3fd864c678aa79ee3d794131b, entries=150, sequenceid=116, filesize=11.6 K 2023-03-26 13:55:46,622 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/a9ab7df802b24ec0b4058ed9cb3ebd59 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/a9ab7df802b24ec0b4058ed9cb3ebd59 2023-03-26 13:55:46,625 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/a9ab7df802b24ec0b4058ed9cb3ebd59, entries=150, sequenceid=116, filesize=11.6 K 2023-03-26 13:55:46,626 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~406.03 KB/415776, currentSize=0 B/0 for 2990228151772e454b24e80da60e1bb2 in 43ms, sequenceid=116, compaction requested=true 2023-03-26 13:55:46,626 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:46,626 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:46,626 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:46,626 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:46,626 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,626 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:46,626 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:46,628 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,628 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,628 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,628 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:46,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:46,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5450944e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:46,630 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:46,630 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,631 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:46,631 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:46,631 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,631 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:46,633 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,633 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:46,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:46,636 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:46,636 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:46,636 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:46,636 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:46,636 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:46,637 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,637 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,637 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,638 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,641 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,643 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,643 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,643 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,643 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,643 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:46,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:46,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:46,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:46,643 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:46,643 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:46,643 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,644 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,644 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,643 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:46,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:46,643 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:46,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:46,743 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:46,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:46,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:46,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:46,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:46,747 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:46,747 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:46,747 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:46,747 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:46,748 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,748 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,749 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:46,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:46,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:46,749 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,750 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,750 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:46,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:46,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:46,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:46,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:46,753 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:46,753 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:46,753 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:46,753 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:46,753 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:46,753 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:46,753 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:46,753 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:46,753 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:46,754 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:46,754 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:46,754 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,754 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:46,754 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:46,757 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,757 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,758 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,758 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:46,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c6df83e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:46,759 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:46,759 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,760 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,760 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:46,760 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:46,760 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:46,760 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,760 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:46,763 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:46,763 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:46,763 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:46,763 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:46,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:46,766 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:46,766 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:46,766 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:46,766 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:46,766 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:46,768 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,768 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,768 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:46,769 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,770 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,778 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,778 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:46,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:46,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:46,778 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:46,778 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,778 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,778 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,778 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,778 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:46,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:46,778 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,778 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:46,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:46,778 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:46,779 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:46,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:46,814 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:55:46,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:46,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:46,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:46,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:46,814 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:46,814 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:46,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679838946828, max timestamp=1679838946843Num uniques 859; 2023-03-26 13:55:46,844 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 13:55:46,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679838946828, max timestamp=1679838946843Num uniques 859; 2023-03-26 13:55:46,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 13:55:46,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679838946828, max timestamp=1679838946843Num uniques 859; 2023-03-26 13:55:46,845 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 13:55:46,879 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:46,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:46,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:46,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:46,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839006869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:46,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839006869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:46,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:46,883 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:46,883 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:46,883 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:46,883 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:46,884 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,884 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,884 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:46,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839006873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:46,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839006877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,885 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:46,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:46,885 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,886 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,886 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:46,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:46,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:46,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839006882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:46,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:46,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:46,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:46,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:46,902 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:46,902 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:46,902 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:46,902 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:46,903 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:46,903 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:46,903 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:46,903 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:46,903 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,905 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:46,905 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:46,909 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,909 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,910 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,910 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:46,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a446e26[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:46,920 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:46,920 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,924 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:46,925 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:46,925 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:46,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:46,926 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:46,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:46,938 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:46,938 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:46,938 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:46,938 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:46,938 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:46,940 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,940 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:46,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,940 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:46,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:46,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:46,941 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:46,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:46,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:46,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,942 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:46,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:46,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:46,945 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:46,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:46,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:46,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:46,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:46,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:46,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:46,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:46,945 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:46,945 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:46,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:46,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:46,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:46,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:46,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839006983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,989 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:46,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839006983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:46,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839006986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:46,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:46,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839006987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839006994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,045 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:47,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:47,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:47,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:47,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:47,048 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:47,048 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:47,049 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:47,049 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:47,049 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,049 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,051 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:47,051 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:47,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:47,052 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,052 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:47,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,052 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:47,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:47,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:47,053 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:47,053 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:47,053 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:47,053 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:47,053 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:47,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:47,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:47,055 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,055 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:47,056 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,056 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:47,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:47,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@371c74c3[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:47,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:47,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,060 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:47,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:47,060 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,061 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:47,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:47,062 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:47,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:47,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:47,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:47,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:47,065 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:47,065 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:47,065 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:47,065 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:47,065 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:47,068 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,068 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,068 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:47,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:47,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:47,075 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:47,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,076 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:47,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:47,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:47,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:47,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:47,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:47,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:47,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:47,079 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:47,079 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:47,078 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:47,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:47,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:47,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,179 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:47,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:47,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:47,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:47,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:47,182 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:47,182 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:47,182 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:47,182 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:47,183 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,183 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,185 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:47,185 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:47,185 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,185 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:47,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:47,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:47,186 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:47,186 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:47,186 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:47,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:47,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:47,187 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:47,187 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:47,187 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:47,187 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:47,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:47,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:47,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:47,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:47,188 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,188 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:47,189 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,189 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:47,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839007190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839007190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:47,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:47,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@673e748a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:47,193 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:47,193 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:47,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:47,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:47,197 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,197 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:47,197 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:47,198 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:47,198 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:47,197 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:47,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:47,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839007194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839007197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:47,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:47,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:47,201 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:47,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:47,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:47,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:47,202 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:47,203 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,203 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:47,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,203 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:47,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:47,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:47,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:47,204 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:47,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:47,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,205 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:47,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839007205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,208 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:47,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:47,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:47,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:47,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:47,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,208 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:47,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:47,208 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:47,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:47,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:47,209 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:47,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:47,226 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/97221351266e4bc181fcfbddab811523 2023-03-26 13:55:47,236 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/01833fa91b44470ab67fdc9f1aa2c551 2023-03-26 13:55:47,245 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=127 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/993aa38b7477426e9c7c6b0284755d3a 2023-03-26 13:55:47,248 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/97221351266e4bc181fcfbddab811523 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/97221351266e4bc181fcfbddab811523 2023-03-26 13:55:47,251 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/97221351266e4bc181fcfbddab811523, entries=250, sequenceid=127, filesize=16.2 K 2023-03-26 13:55:47,252 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/01833fa91b44470ab67fdc9f1aa2c551 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/01833fa91b44470ab67fdc9f1aa2c551 2023-03-26 13:55:47,254 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/01833fa91b44470ab67fdc9f1aa2c551, entries=150, sequenceid=127, filesize=11.6 K 2023-03-26 13:55:47,255 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/993aa38b7477426e9c7c6b0284755d3a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/993aa38b7477426e9c7c6b0284755d3a 2023-03-26 13:55:47,258 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/993aa38b7477426e9c7c6b0284755d3a, entries=150, sequenceid=127, filesize=11.6 K 2023-03-26 13:55:47,258 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 2990228151772e454b24e80da60e1bb2 in 445ms, sequenceid=127, compaction requested=true 2023-03-26 13:55:47,258 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:47,258 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:47,259 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:47,259 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:47,259 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:47,259 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:47,259 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:47,259 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/B is initiating minor compaction (all files) 2023-03-26 13:55:47,260 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/B in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:47,260 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/d8bdfb87dcde472c818669b1029cd136, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/10b9c1d3fd864c678aa79ee3d794131b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/01833fa91b44470ab67fdc9f1aa2c551] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=35.0 K 2023-03-26 13:55:47,260 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting d8bdfb87dcde472c818669b1029cd136, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1679838945611 2023-03-26 13:55:47,260 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40544 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:47,260 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/A is initiating minor compaction (all files) 2023-03-26 13:55:47,260 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 10b9c1d3fd864c678aa79ee3d794131b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1679838945668 2023-03-26 13:55:47,260 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/A in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:47,261 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/dfcef5bee5344e3793df5f5f741ba2fe, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/d199f1349acc4b7abf4b0e135df92383, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/97221351266e4bc181fcfbddab811523] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=39.6 K 2023-03-26 13:55:47,261 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 01833fa91b44470ab67fdc9f1aa2c551, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1679838946807 2023-03-26 13:55:47,261 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting dfcef5bee5344e3793df5f5f741ba2fe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1679838945611 2023-03-26 13:55:47,261 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting d199f1349acc4b7abf4b0e135df92383, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1679838945668 2023-03-26 13:55:47,261 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 97221351266e4bc181fcfbddab811523, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1679838946797 2023-03-26 13:55:47,266 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#B#compaction#464 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:47,267 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#A#compaction#465 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:47,277 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/743f7c3699b444a3875d6c0d1a0970e5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/743f7c3699b444a3875d6c0d1a0970e5 2023-03-26 13:55:47,281 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/B of 2990228151772e454b24e80da60e1bb2 into 743f7c3699b444a3875d6c0d1a0970e5(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:47,281 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:47,281 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/B, priority=13, startTime=1679838947259; duration=0sec 2023-03-26 13:55:47,281 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:47,281 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:55:47,282 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47682 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:55:47,282 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/C is initiating minor compaction (all files) 2023-03-26 13:55:47,283 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/C in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:47,283 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c0e3e1b29bf14b999d52f01f0e0529bf, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/2d5639a9a1054f04a9fded3472d1b7ce, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/a9ab7df802b24ec0b4058ed9cb3ebd59, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/993aa38b7477426e9c7c6b0284755d3a] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=46.6 K 2023-03-26 13:55:47,283 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting c0e3e1b29bf14b999d52f01f0e0529bf, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1679838944992 2023-03-26 13:55:47,283 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 2d5639a9a1054f04a9fded3472d1b7ce, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1679838945611 2023-03-26 13:55:47,284 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting a9ab7df802b24ec0b4058ed9cb3ebd59, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=116, earliestPutTs=1679838945668 2023-03-26 13:55:47,284 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 993aa38b7477426e9c7c6b0284755d3a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1679838946807 2023-03-26 13:55:47,290 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#C#compaction#466 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:47,309 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:47,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:47,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:47,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:47,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:47,312 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:47,312 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:47,312 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:47,312 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:47,312 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,312 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,314 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:47,315 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:47,315 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,315 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:47,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:47,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:47,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:47,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:47,316 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:47,316 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:47,316 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:47,316 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:47,316 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:47,316 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:47,316 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:47,317 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-26 13:55:47,317 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:47,317 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:47,317 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:47,317 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:47,317 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:47,317 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:47,325 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/b079d71a53a743e38b2f618c42199844 2023-03-26 13:55:47,335 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/9d6a2b188a6f4f52a46d83f464a67f0d 2023-03-26 13:55:47,429 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-26 13:55:47,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:47,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:47,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839007504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839007505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,513 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839007506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,514 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839007507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,518 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839007512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839007613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839007614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,620 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839007614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839007619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,685 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/a7f7ce591e3b49a1b63620485392bcd4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/a7f7ce591e3b49a1b63620485392bcd4 2023-03-26 13:55:47,692 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/A of 2990228151772e454b24e80da60e1bb2 into a7f7ce591e3b49a1b63620485392bcd4(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:47,692 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:47,692 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/A, priority=13, startTime=1679838947258; duration=0sec 2023-03-26 13:55:47,692 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:47,700 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/bf0131b798cf471aa9961ab4f57ad064 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/bf0131b798cf471aa9961ab4f57ad064 2023-03-26 13:55:47,704 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2990228151772e454b24e80da60e1bb2/C of 2990228151772e454b24e80da60e1bb2 into bf0131b798cf471aa9961ab4f57ad064(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:47,704 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:47,704 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/C, priority=12, startTime=1679838947259; duration=0sec 2023-03-26 13:55:47,704 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:47,745 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/b1b9845fc1254fecb7713f7b5476283c 2023-03-26 13:55:47,750 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/b079d71a53a743e38b2f618c42199844 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b079d71a53a743e38b2f618c42199844 2023-03-26 13:55:47,753 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b079d71a53a743e38b2f618c42199844, entries=150, sequenceid=163, filesize=11.8 K 2023-03-26 13:55:47,754 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/9d6a2b188a6f4f52a46d83f464a67f0d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9d6a2b188a6f4f52a46d83f464a67f0d 2023-03-26 13:55:47,757 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9d6a2b188a6f4f52a46d83f464a67f0d, entries=150, sequenceid=163, filesize=11.8 K 2023-03-26 13:55:47,758 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/b1b9845fc1254fecb7713f7b5476283c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b1b9845fc1254fecb7713f7b5476283c 2023-03-26 13:55:47,765 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b1b9845fc1254fecb7713f7b5476283c, entries=150, sequenceid=163, filesize=11.8 K 2023-03-26 13:55:47,765 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 2990228151772e454b24e80da60e1bb2 in 448ms, sequenceid=163, compaction requested=false 2023-03-26 13:55:47,766 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:47,766 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:47,766 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:47,766 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:47,766 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,766 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:47,766 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:47,767 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,767 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:47,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:47,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:47,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6411dc5b[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:47,769 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:47,769 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,770 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,770 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:47,770 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:47,770 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:47,770 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,770 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:47,775 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,775 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:47,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:47,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:47,775 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:47,775 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:47,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:47,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:47,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:47,777 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:47,777 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:47,777 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:47,777 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:47,777 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:47,779 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,779 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:47,779 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:47,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:47,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,780 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,781 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,784 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:47,784 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:47,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:47,784 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:47,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:47,784 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:47,784 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,784 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,784 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,784 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,784 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,784 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:47,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:47,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:47,784 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,785 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:47,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:47,823 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 13:55:47,823 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:47,823 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:47,823 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:47,823 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:47,823 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:47,823 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:47,834 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/66fbc2cf0b4f42b2a286a48bca001115 2023-03-26 13:55:47,843 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/2c092c7a972941038ce0a6b3692b9603 2023-03-26 13:55:47,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838947844, max timestamp=1679838947857Num uniques 859; 2023-03-26 13:55:47,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:55:47,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838947844, max timestamp=1679838947857Num uniques 859; 2023-03-26 13:55:47,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838947844, max timestamp=1679838947857Num uniques 859; 2023-03-26 13:55:47,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:55:47,859 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:55:47,863 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/6a45cf1284004427afbbd3b143f5248d 2023-03-26 13:55:47,865 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/66fbc2cf0b4f42b2a286a48bca001115 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/66fbc2cf0b4f42b2a286a48bca001115 2023-03-26 13:55:47,868 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/66fbc2cf0b4f42b2a286a48bca001115, entries=200, sequenceid=176, filesize=14.1 K 2023-03-26 13:55:47,869 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/2c092c7a972941038ce0a6b3692b9603 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/2c092c7a972941038ce0a6b3692b9603 2023-03-26 13:55:47,872 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/2c092c7a972941038ce0a6b3692b9603, entries=150, sequenceid=176, filesize=11.8 K 2023-03-26 13:55:47,873 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/6a45cf1284004427afbbd3b143f5248d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/6a45cf1284004427afbbd3b143f5248d 2023-03-26 13:55:47,876 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/6a45cf1284004427afbbd3b143f5248d, entries=150, sequenceid=176, filesize=11.8 K 2023-03-26 13:55:47,877 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=93.93 KB/96180 for 2990228151772e454b24e80da60e1bb2 in 54ms, sequenceid=176, compaction requested=true 2023-03-26 13:55:47,877 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:47,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:47,877 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:47,877 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:47,878 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:47,878 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-26 13:55:47,878 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=100.63 KB heapSize=276 KB 2023-03-26 13:55:47,878 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:47,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:47,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:47,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:47,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:47,879 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:47,879 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:47,884 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36266 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:47,884 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38656 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:47,884 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/A is initiating minor compaction (all files) 2023-03-26 13:55:47,884 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/B is initiating minor compaction (all files) 2023-03-26 13:55:47,884 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/B in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:47,885 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/743f7c3699b444a3875d6c0d1a0970e5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9d6a2b188a6f4f52a46d83f464a67f0d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/2c092c7a972941038ce0a6b3692b9603] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=35.4 K 2023-03-26 13:55:47,884 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/A in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:47,885 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/a7f7ce591e3b49a1b63620485392bcd4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b079d71a53a743e38b2f618c42199844, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/66fbc2cf0b4f42b2a286a48bca001115] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=37.8 K 2023-03-26 13:55:47,885 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:47,885 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 743f7c3699b444a3875d6c0d1a0970e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1679838946807 2023-03-26 13:55:47,885 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting a7f7ce591e3b49a1b63620485392bcd4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1679838946807 2023-03-26 13:55:47,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:47,886 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 9d6a2b188a6f4f52a46d83f464a67f0d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1679838946867 2023-03-26 13:55:47,887 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting b079d71a53a743e38b2f618c42199844, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1679838946867 2023-03-26 13:55:47,887 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 2c092c7a972941038ce0a6b3692b9603, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1679838947505 2023-03-26 13:55:47,887 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 66fbc2cf0b4f42b2a286a48bca001115, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1679838947503 2023-03-26 13:55:47,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:47,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:47,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:47,901 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:47,901 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:47,901 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:47,901 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:47,901 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,901 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,901 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#A#compaction#474 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:47,902 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#B#compaction#475 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:47,903 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:47,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:47,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:47,903 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,904 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,904 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:47,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:47,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:47,908 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/d727767162634314ab67f7fb0affa56e 2023-03-26 13:55:47,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:47,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:47,908 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:47,908 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:47,908 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:47,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839007904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839007904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839007905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:47,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839007906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,916 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:47,916 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:47,916 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:47,916 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:47,916 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:47,916 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:47,916 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:47,916 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:47,916 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,917 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:47,917 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:47,918 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,918 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:47,918 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,918 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:47,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:47,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d7daba1[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:47,920 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:47,920 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,921 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/db268056546f4b4d96fe3502dc512a2d 2023-03-26 13:55:47,922 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,922 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:47,922 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:47,922 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:47,923 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,923 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:47,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:47,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:47,925 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:47,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:47,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:47,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:47,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:47,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:47,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:47,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:47,929 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:47,929 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:47,929 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:47,929 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:47,929 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:47,930 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=207 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/c518bd17bfaf41f784ab95b083e09baf 2023-03-26 13:55:47,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:47,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:47,931 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:47,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:47,932 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:47,933 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/d727767162634314ab67f7fb0affa56e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/d727767162634314ab67f7fb0affa56e 2023-03-26 13:55:47,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:47,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:47,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:47,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:47,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:47,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:47,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:47,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:47,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:47,935 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:47,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:47,935 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:47,935 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:47,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:47,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:47,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:47,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:47,936 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/d727767162634314ab67f7fb0affa56e, entries=250, sequenceid=207, filesize=16.4 K 2023-03-26 13:55:47,937 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/db268056546f4b4d96fe3502dc512a2d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/db268056546f4b4d96fe3502dc512a2d 2023-03-26 13:55:47,940 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/db268056546f4b4d96fe3502dc512a2d, entries=150, sequenceid=207, filesize=11.8 K 2023-03-26 13:55:47,941 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/c518bd17bfaf41f784ab95b083e09baf as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c518bd17bfaf41f784ab95b083e09baf 2023-03-26 13:55:47,943 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c518bd17bfaf41f784ab95b083e09baf, entries=150, sequenceid=207, filesize=11.8 K 2023-03-26 13:55:47,944 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~295.88 KB/302976, currentSize=73.80 KB/75570 for 2990228151772e454b24e80da60e1bb2 in 66ms, sequenceid=207, compaction requested=true 2023-03-26 13:55:47,944 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:47,944 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-26 13:55:47,944 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-26 13:55:47,944 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-26 13:55:48,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:48,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-26 13:55:48,017 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:48,018 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:48,018 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:48,018 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:48,018 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:48,018 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:48,035 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:48,035 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839008029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839008030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:48,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839008032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:48,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:48,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:48,038 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,038 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:48,038 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:48,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839008032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,038 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:48,038 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:48,039 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,039 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,040 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,040 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:48,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:48,040 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,040 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:48,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:48,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:48,042 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:48,042 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:48,042 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:48,042 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:48,042 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:48,042 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:48,042 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:48,042 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:48,042 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,042 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:48,042 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:48,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839008035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,044 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,044 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,044 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,044 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:48,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69936aa1[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:48,046 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:48,046 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,047 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,047 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:48,047 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:48,047 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:48,047 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,047 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:48,048 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,048 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:48,048 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,048 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:48,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:48,051 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:48,051 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:48,051 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:48,051 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:48,051 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:48,052 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,052 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,052 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:48,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:48,053 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,054 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,057 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:48,057 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:48,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:48,057 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:48,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:48,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:48,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:48,058 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:48,058 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:48,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:48,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:48,058 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,058 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,058 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,058 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,058 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,058 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,058 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839008136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839008137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839008137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839008139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839008143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,158 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:48,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:48,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:48,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:48,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:48,161 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:48,161 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:48,161 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:48,161 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:48,161 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,161 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,163 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,163 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:48,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:48,163 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,163 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:48,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:48,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:48,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:48,164 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:48,164 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:48,164 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:48,165 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:48,165 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:48,165 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:48,165 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:48,167 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:48,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d9dc93d[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:48,169 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:48,169 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,170 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,170 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:48,170 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:48,170 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:48,170 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,170 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:48,175 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:48,176 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:48,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:48,177 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:48,177 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:48,178 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:48,178 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:48,178 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:48,179 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,179 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,179 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:48,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,179 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:48,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,182 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,188 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:48,188 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:48,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:48,188 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:48,188 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:48,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:48,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:48,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:48,188 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:48,188 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:48,188 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,188 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:48,188 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,188 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,188 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,188 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,288 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:48,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:48,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:48,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:48,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:48,291 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:48,291 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:48,291 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:48,291 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:48,292 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,292 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,293 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,293 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:48,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:48,293 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,293 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:48,294 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:48,294 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:48,295 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:48,295 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:48,295 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:48,295 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:48,295 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:48,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,297 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,297 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,297 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:48,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@170431d1[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:48,300 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:48,300 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,302 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,302 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:48,302 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:48,302 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:48,302 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,302 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:48,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:48,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:48,304 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:48,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:48,307 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:48,307 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:48,307 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:48,307 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:48,307 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:48,309 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,309 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,309 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:48,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,312 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,315 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,315 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:48,315 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:48,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:48,315 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:48,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:48,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:48,316 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:48,316 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:48,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:48,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:48,315 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:48,315 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,316 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,316 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,316 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,316 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:48,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:48,317 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/eebd0bf91a3d42fb905a2e3fa4eb3ad5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/eebd0bf91a3d42fb905a2e3fa4eb3ad5 2023-03-26 13:55:48,321 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/e05205493a1943ceb60f5a1c1b291791 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/e05205493a1943ceb60f5a1c1b291791 2023-03-26 13:55:48,322 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/A of 2990228151772e454b24e80da60e1bb2 into eebd0bf91a3d42fb905a2e3fa4eb3ad5(size=12.2 K), total size for store is 28.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:48,322 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:48,322 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/A, priority=13, startTime=1679838947877; duration=0sec 2023-03-26 13:55:48,322 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-03-26 13:55:48,323 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:55:48,324 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48268 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:55:48,324 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/C is initiating minor compaction (all files) 2023-03-26 13:55:48,324 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/C in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:48,324 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/bf0131b798cf471aa9961ab4f57ad064, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b1b9845fc1254fecb7713f7b5476283c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/6a45cf1284004427afbbd3b143f5248d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c518bd17bfaf41f784ab95b083e09baf] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=47.1 K 2023-03-26 13:55:48,324 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting bf0131b798cf471aa9961ab4f57ad064, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=127, earliestPutTs=1679838946807 2023-03-26 13:55:48,325 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting b1b9845fc1254fecb7713f7b5476283c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1679838946867 2023-03-26 13:55:48,325 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 6a45cf1284004427afbbd3b143f5248d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1679838947505 2023-03-26 13:55:48,325 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/B of 2990228151772e454b24e80da60e1bb2 into e05205493a1943ceb60f5a1c1b291791(size=12.2 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:48,325 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting c518bd17bfaf41f784ab95b083e09baf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1679838947868 2023-03-26 13:55:48,325 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:48,325 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/B, priority=13, startTime=1679838947878; duration=0sec 2023-03-26 13:55:48,326 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-26 13:55:48,326 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 4 compacting, 0 eligible, 16 blocking 2023-03-26 13:55:48,326 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 13:55:48,326 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 13:55:48,326 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. because compaction request was cancelled 2023-03-26 13:55:48,326 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-26 13:55:48,327 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 13:55:48,327 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 13:55:48,327 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. because compaction request was cancelled 2023-03-26 13:55:48,327 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-26 13:55:48,328 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 13:55:48,328 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 13:55:48,328 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. because compaction request was cancelled 2023-03-26 13:55:48,332 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#C#compaction#479 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:48,339 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/c788d45f7f4344649f7b01404cc60af1 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c788d45f7f4344649f7b01404cc60af1 2023-03-26 13:55:48,342 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2990228151772e454b24e80da60e1bb2/C of 2990228151772e454b24e80da60e1bb2 into c788d45f7f4344649f7b01404cc60af1(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:48,342 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:48,342 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/C, priority=12, startTime=1679838947944; duration=0sec 2023-03-26 13:55:48,343 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:48,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839008339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839008339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839008340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839008345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839008345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,416 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:48,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:48,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:48,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:48,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:48,419 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:48,419 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:48,419 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:48,419 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:48,420 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,420 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,422 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,422 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:48,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:48,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:48,422 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,422 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:48,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:48,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:48,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:48,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:48,423 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:48,423 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:48,423 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:48,423 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:48,423 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:48,424 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:48,424 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:48,424 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:48,424 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:48,424 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:48,424 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:48,424 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,424 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:48,424 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:48,425 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/7b6ccc3103da409b8e376cd1d38c82ac 2023-03-26 13:55:48,425 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,425 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:48,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4774b033[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:48,427 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:48,427 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,428 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,429 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:48,429 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:48,429 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:48,429 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,429 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:48,430 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,430 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:48,430 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:48,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,430 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:48,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:48,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:48,433 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:48,433 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:48,433 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:48,433 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:48,433 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:48,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:48,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:48,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,437 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,439 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:48,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:48,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:48,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:48,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:48,441 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:48,441 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:48,441 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:48,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:48,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:48,541 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:48,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:48,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:48,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:48,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:48,544 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:48,544 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:48,545 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:48,545 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:48,545 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,545 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,546 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:48,546 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:48,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:48,547 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,547 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:48,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:48,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:48,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:48,548 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:48,548 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:48,548 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:48,548 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:48,548 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:48,548 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:48,548 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:48,548 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:48,548 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:48,548 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:48,548 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:48,548 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,548 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:48,549 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:48,550 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,550 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,550 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,550 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:48,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35c01c1e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:48,552 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:48,552 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,554 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,554 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:48,554 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,554 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:48,554 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:48,554 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:48,555 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,555 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:48,555 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,555 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:48,557 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:48,557 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:48,558 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:48,558 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:48,558 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:48,558 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:48,559 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,559 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,559 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:48,560 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,562 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:48,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:48,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:48,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,567 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:48,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:48,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:48,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:48,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:48,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,567 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:48,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,567 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:48,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,567 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839008644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839008647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839008647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839008649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:48,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839008651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,667 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:48,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:48,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:48,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:48,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:48,671 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:48,671 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:48,671 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:48,671 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:48,671 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,671 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,672 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:48,672 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:48,673 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,673 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:48,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:48,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:48,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:48,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:48,674 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:48,674 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:48,674 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:48,674 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:48,674 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:48,674 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:48,674 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:48,675 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:48,675 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:48,675 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:48,675 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:48,675 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,675 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:48,675 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:48,676 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,676 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,677 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,677 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:48,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2687efba[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:48,678 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:48,678 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,680 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:48,681 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,681 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:48,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:48,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:48,682 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:48,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:48,685 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:48,685 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:48,685 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:48,685 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:48,685 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:48,687 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,688 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,689 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,689 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:48,689 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:48,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:48,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:48,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:48,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:48,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:48,691 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:48,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:48,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:48,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,691 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:48,696 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:48,696 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:48,796 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:48,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:48,798 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:48,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:48,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:48,799 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:48,799 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:48,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:48,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:48,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,802 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,802 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:48,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:48,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:48,802 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,802 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:48,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:48,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:48,803 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:48,803 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:48,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:48,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:48,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:48,804 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:48,804 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:48,804 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:48,804 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:48,804 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:48,804 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:48,804 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:48,804 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:48,804 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,804 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:48,804 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:48,806 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,806 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,806 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,806 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:48,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@103bc802[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:48,807 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:48,807 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,809 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,809 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:48,809 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:48,809 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:48,809 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,809 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:48,810 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,810 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:48,810 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,810 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:48,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:48,812 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:48,812 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:48,812 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:48,812 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:48,812 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:48,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:48,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:48,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,816 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,818 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,818 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,818 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,818 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,818 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,818 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:48,818 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:48,818 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:48,818 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,818 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:48,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:48,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:48,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:48,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:48,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:48,818 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:48,818 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:48,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:48,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:48,838 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/88d3d5248b884ff7a7e75a9cb37a8323 2023-03-26 13:55:48,846 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/b04e3dc6d46949b793e1a8a9d37bb931 2023-03-26 13:55:48,849 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/7b6ccc3103da409b8e376cd1d38c82ac as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/7b6ccc3103da409b8e376cd1d38c82ac 2023-03-26 13:55:48,852 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/7b6ccc3103da409b8e376cd1d38c82ac, entries=250, sequenceid=223, filesize=16.4 K 2023-03-26 13:55:48,853 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/88d3d5248b884ff7a7e75a9cb37a8323 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/88d3d5248b884ff7a7e75a9cb37a8323 2023-03-26 13:55:48,855 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/88d3d5248b884ff7a7e75a9cb37a8323, entries=150, sequenceid=223, filesize=11.8 K 2023-03-26 13:55:48,856 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/b04e3dc6d46949b793e1a8a9d37bb931 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b04e3dc6d46949b793e1a8a9d37bb931 2023-03-26 13:55:48,859 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b04e3dc6d46949b793e1a8a9d37bb931, entries=150, sequenceid=223, filesize=11.8 K 2023-03-26 13:55:48,859 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 2990228151772e454b24e80da60e1bb2 in 843ms, sequenceid=223, compaction requested=true 2023-03-26 13:55:48,859 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:48,860 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:48,860 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:48,860 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:48,860 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:48,860 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:48,861 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 46078 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:48,862 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:48,862 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/A is initiating minor compaction (all files) 2023-03-26 13:55:48,862 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/B is initiating minor compaction (all files) 2023-03-26 13:55:48,862 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/A in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:48,862 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/B in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:48,862 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/eebd0bf91a3d42fb905a2e3fa4eb3ad5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/d727767162634314ab67f7fb0affa56e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/7b6ccc3103da409b8e376cd1d38c82ac] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=45.0 K 2023-03-26 13:55:48,862 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/e05205493a1943ceb60f5a1c1b291791, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/db268056546f4b4d96fe3502dc512a2d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/88d3d5248b884ff7a7e75a9cb37a8323] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=35.7 K 2023-03-26 13:55:48,862 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting eebd0bf91a3d42fb905a2e3fa4eb3ad5, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1679838947505 2023-03-26 13:55:48,862 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e05205493a1943ceb60f5a1c1b291791, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1679838947505 2023-03-26 13:55:48,863 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting d727767162634314ab67f7fb0affa56e, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1679838947868 2023-03-26 13:55:48,863 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting db268056546f4b4d96fe3502dc512a2d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1679838947868 2023-03-26 13:55:48,863 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 7b6ccc3103da409b8e376cd1d38c82ac, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1679838947891 2023-03-26 13:55:48,863 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 88d3d5248b884ff7a7e75a9cb37a8323, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1679838947891 2023-03-26 13:55:48,868 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#A#compaction#482 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:48,869 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#B#compaction#483 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:48,876 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/49fb33e7f1034d10ab91850b02b55422 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/49fb33e7f1034d10ab91850b02b55422 2023-03-26 13:55:48,879 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/19be7a99b67c4cb283b2208717809e1b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/19be7a99b67c4cb283b2208717809e1b 2023-03-26 13:55:48,880 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/A of 2990228151772e454b24e80da60e1bb2 into 49fb33e7f1034d10ab91850b02b55422(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:48,880 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:48,880 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/A, priority=13, startTime=1679838948860; duration=0sec 2023-03-26 13:55:48,880 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:48,880 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-26 13:55:48,881 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 13:55:48,881 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 13:55:48,881 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. because compaction request was cancelled 2023-03-26 13:55:48,883 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/B of 2990228151772e454b24e80da60e1bb2 into 19be7a99b67c4cb283b2208717809e1b(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:48,883 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:48,883 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/B, priority=13, startTime=1679838948860; duration=0sec 2023-03-26 13:55:48,883 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:48,919 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:48,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:48,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:48,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:48,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:48,922 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:48,922 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:48,922 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:48,922 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:48,922 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,922 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,924 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:48,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:48,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:48,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:48,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:48,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:48,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:48,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:48,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:48,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:48,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:48,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:48,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:48,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:48,926 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:48,926 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:48,926 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-26 13:55:48,926 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:48,926 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:48,926 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:48,926 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:48,926 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:48,926 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:49,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:49,155 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:49,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839009169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,178 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839009173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839009173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839009174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839009177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839009278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839009280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839009280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839009280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839009281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,333 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/51a5ffda71ce49339fc158ca8f0f31a1 2023-03-26 13:55:49,341 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/9a86e8fa9a20488e8152851a2643c1b5 2023-03-26 13:55:49,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839009484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839009484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839009486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839009485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839009488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,748 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/8e04ee371b9a49baba1f4ee2f6299b3d 2023-03-26 13:55:49,752 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/51a5ffda71ce49339fc158ca8f0f31a1 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/51a5ffda71ce49339fc158ca8f0f31a1 2023-03-26 13:55:49,755 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/51a5ffda71ce49339fc158ca8f0f31a1, entries=150, sequenceid=244, filesize=11.8 K 2023-03-26 13:55:49,756 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/9a86e8fa9a20488e8152851a2643c1b5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9a86e8fa9a20488e8152851a2643c1b5 2023-03-26 13:55:49,760 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9a86e8fa9a20488e8152851a2643c1b5, entries=150, sequenceid=244, filesize=11.8 K 2023-03-26 13:55:49,760 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/8e04ee371b9a49baba1f4ee2f6299b3d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/8e04ee371b9a49baba1f4ee2f6299b3d 2023-03-26 13:55:49,763 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/8e04ee371b9a49baba1f4ee2f6299b3d, entries=150, sequenceid=244, filesize=11.8 K 2023-03-26 13:55:49,764 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 2990228151772e454b24e80da60e1bb2 in 838ms, sequenceid=244, compaction requested=true 2023-03-26 13:55:49,764 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:49,764 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:49,764 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:49,766 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,766 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:49,766 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:49,766 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:49,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:49,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@722c071e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:49,767 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:49,767 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:49,768 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:49,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:49,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:49,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:49,769 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:49,769 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,769 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:49,771 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:49,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,771 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:49,771 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:49,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:49,773 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:49,773 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:49,773 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:49,773 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:49,773 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:49,775 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:49,775 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:49,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:49,775 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:49,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:49,776 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:49,777 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,779 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,779 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:49,779 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:49,779 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:49,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:49,779 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:49,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:49,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:49,779 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:49,779 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:49,779 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:49,779 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:49,779 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:49,779 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:49,779 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:49,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:49,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:49,794 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-26 13:55:49,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:49,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:49,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:49,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:49,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:49,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:49,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839009802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839009803, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839009806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839009807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839009807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,880 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:49,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:49,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:49,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:49,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:49,885 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:49,885 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:49,885 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:49,885 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:49,886 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:49,886 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:49,891 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:49,891 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:49,892 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,892 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:49,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:49,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:49,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:49,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:49,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:49,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:49,893 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:49,893 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:49,893 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:49,896 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:49,896 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:49,896 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:49,896 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:49,896 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:49,896 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:49,896 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:49,896 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:49,896 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,896 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:49,896 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:49,898 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:49,898 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:49,898 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:49,904 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4739c5a9[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:49,906 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:49,906 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:49,907 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:49,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:49,907 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,907 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:49,907 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:49,910 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:49,910 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:49,911 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:49,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:49,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,914 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:49,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839009910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:49,914 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:49,914 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:49,914 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:49,915 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:49,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839009910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839009910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,917 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:49,917 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:49,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:49,917 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:49,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:49,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:49,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:49,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839009914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:49,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:49,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839009914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,920 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:49,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:49,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:49,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,925 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:49,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:49,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,927 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,927 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:49,927 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:49,927 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:49,927 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:49,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:49,927 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:49,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:49,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:49,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:49,927 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:49,928 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:49,928 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:49,927 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:49,928 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:49,928 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:49,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:49,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:50,028 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:50,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:50,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:50,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:50,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:50,031 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:50,031 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:50,031 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:50,031 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:50,031 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,032 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,033 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:50,033 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:50,033 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,033 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:50,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:50,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:50,034 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:50,034 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:50,034 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:50,034 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:50,034 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:50,034 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:50,034 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:50,035 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:50,035 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:50,035 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:50,035 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:50,035 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,035 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:50,035 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:50,039 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,039 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,040 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,040 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:50,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e5a08cb[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:50,041 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:50,041 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,042 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,042 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:50,042 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:50,042 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:50,043 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,043 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:50,043 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:50,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:50,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:50,045 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:50,046 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:50,046 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:50,046 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:50,046 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:50,047 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,047 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,047 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,048 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,052 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,052 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,052 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:50,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:50,052 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:50,052 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,052 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,052 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,052 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,052 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:50,052 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:50,060 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:50,060 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:50,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:50,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:50,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:50,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:50,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:50,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839010116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:50,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839010116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:50,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839010117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:50,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839010121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:50,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839010125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,153 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:50,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:50,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:50,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:50,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:50,156 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:50,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:50,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:50,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:50,157 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,157 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:50,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:50,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:50,158 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,159 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,159 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,160 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:50,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:50,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:50,160 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:50,160 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:50,160 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:50,160 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:50,160 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:50,160 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:50,160 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:50,160 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:50,160 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:50,161 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:50,161 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:50,161 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,161 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:50,161 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:50,162 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,162 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,162 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,162 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:50,164 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@106fb0da[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:50,164 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:50,164 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,165 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,165 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:50,165 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:50,165 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:50,166 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,166 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:50,168 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,168 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:50,168 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:50,168 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:50,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:50,170 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:50,170 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:50,170 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:50,170 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:50,170 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:50,172 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,172 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,172 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:50,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:50,173 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,174 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,176 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:50,176 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:50,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:50,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:50,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:50,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:50,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:50,176 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:50,176 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,176 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,176 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,176 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:50,177 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:50,176 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,177 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,177 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:50,177 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:50,211 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/ff219c2625db4c1f9e7845bd4f5eb1cd 2023-03-26 13:55:50,277 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:50,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:50,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:50,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:50,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:50,280 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:50,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:50,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:50,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:50,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,281 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:50,281 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:50,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:50,282 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,282 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:50,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:50,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:50,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:50,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:50,283 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:50,283 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:50,283 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:50,283 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:50,284 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:50,286 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,286 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,286 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,286 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:50,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@269413e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:50,289 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:50,289 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,292 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,292 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:50,292 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:50,292 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:50,292 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,292 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:50,293 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,294 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:50,294 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:50,294 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:50,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:50,296 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:50,296 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:50,296 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:50,296 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:50,296 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:50,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,298 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,300 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,302 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,302 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:50,302 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:50,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:50,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:50,302 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:50,303 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:50,302 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,303 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,303 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,303 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,303 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:50,307 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:50,307 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:50,408 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:50,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:50,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:50,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:50,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:50,411 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:50,411 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:50,411 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:50,411 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:50,411 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,412 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,414 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:50,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:50,414 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,415 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,415 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:50,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:50,417 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:50,417 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:50,417 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:50,417 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:50,417 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:50,417 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:50,417 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:50,419 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,419 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,419 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,419 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:50,421 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bf4c857[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:50,421 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:50,421 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,423 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,424 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:50,424 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:50,424 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,424 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:50,424 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:50,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:50,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839010421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,425 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,425 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:50,426 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,426 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:50,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839010423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,427 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:50,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839010423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:50,429 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:50,429 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:50,429 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:50,429 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:50,429 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:50,429 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:50,433 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,433 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:50,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:50,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:50,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839010428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:50,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839010428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,439 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,440 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,441 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,443 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,443 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,443 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:50,443 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:50,443 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:50,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:50,443 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,443 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:50,443 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:50,444 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:50,444 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:50,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:50,443 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,444 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:50,544 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:50,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:50,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:50,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:50,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:50,554 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:50,555 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:50,555 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:50,555 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:50,556 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,556 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,557 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:50,557 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:50,557 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,557 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:50,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,558 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:50,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:50,560 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:50,560 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:50,560 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:50,564 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:50,564 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:50,564 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:50,564 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:50,564 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:50,564 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:50,564 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:50,564 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:50,564 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,564 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:50,564 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:50,566 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,566 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,566 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,566 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:50,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e685b2[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:50,567 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:50,567 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,569 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:50,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:50,570 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:50,570 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,570 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:50,571 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:50,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:50,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:50,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:50,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:50,574 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:50,574 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:50,574 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:50,574 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:50,574 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:50,575 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,575 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,575 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:50,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,576 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:50,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:50,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,577 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,579 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:50,579 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:50,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:50,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:50,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:50,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:50,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,580 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:50,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,580 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:50,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:50,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:50,580 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:50,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,580 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,628 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/33b5fb4caa5743cca9f3bc2040e652c9 2023-03-26 13:55:50,680 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:50,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:50,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:50,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:50,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:50,685 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:50,685 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:50,685 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:50,685 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:50,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,686 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,688 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:50,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:50,688 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,688 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,688 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:50,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:50,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:50,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:50,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:50,692 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:50,692 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:50,692 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:50,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:50,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:50,693 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:50,693 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:50,693 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:50,693 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:50,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:50,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:50,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:50,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:50,695 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:50,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68e20cb4[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:50,698 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:50,698 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,699 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,699 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:50,699 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:50,699 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:50,700 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,700 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:50,705 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:50,705 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:50,705 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,705 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:50,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:50,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:50,709 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:50,710 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:50,710 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:50,710 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:50,710 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:50,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:50,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,713 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,714 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:50,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:50,717 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:50,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:50,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:50,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:50,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:50,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:50,718 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:50,718 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:50,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,718 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,718 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,718 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,818 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:50,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:50,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:50,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:50,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:50,821 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:50,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:50,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:50,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:50,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,822 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,823 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,823 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:50,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:50,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:50,823 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,823 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:50,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:50,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:50,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:50,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:50,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:50,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:50,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:50,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:50,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:50,824 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:50,825 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:50,825 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:50,825 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:50,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:50,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:50,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:50,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:50,828 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,828 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,828 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,828 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:50,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:50,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@204ac027[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:50,829 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:50,829 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,830 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,830 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:50,830 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:50,830 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:50,831 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,831 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:50,832 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,832 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,832 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:50,832 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:50,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:50,834 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:50,834 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:50,834 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:50,834 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:50,834 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:50,835 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,835 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,836 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,836 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,838 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,841 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,841 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,841 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:50,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:50,841 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:50,841 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:50,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:50,841 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:50,841 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,841 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:50,842 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:50,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:50,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:50,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839010927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:50,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839010928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:50,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839010930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:50,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839010937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,942 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:50,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:50,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:50,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839010938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:50,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:50,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:50,945 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:50,945 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:50,945 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:50,945 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:50,945 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,945 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,946 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:50,946 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:50,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:50,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:50,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:50,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:50,948 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:50,948 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:50,948 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:50,948 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:50,948 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:50,948 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:50,948 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:50,949 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:50,949 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:50,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:50,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:50,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:50,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:50,950 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,950 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,950 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,950 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:50,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d510209[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:50,952 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:50,952 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,953 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,953 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:50,953 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:50,953 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:50,953 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,953 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:50,955 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,955 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,955 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:50,955 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:50,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:50,957 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:50,957 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:50,957 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:50,957 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:50,957 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:50,959 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,959 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:50,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,959 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:50,959 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:50,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,960 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,962 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:50,962 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:50,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:50,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:50,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:50,962 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,962 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,962 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:50,962 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:50,962 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,962 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:50,962 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:50,962 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:50,962 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:50,962 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:50,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:50,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:51,062 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/1da4573419274ad29a3f54dc49152257 2023-03-26 13:55:51,062 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:51,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:51,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:51,065 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/ff219c2625db4c1f9e7845bd4f5eb1cd as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/ff219c2625db4c1f9e7845bd4f5eb1cd 2023-03-26 13:55:51,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:51,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:51,066 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:51,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:51,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:51,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:51,067 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,067 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,069 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:51,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:51,069 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,069 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,069 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:51,069 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:51,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,070 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:51,070 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:51,070 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:51,070 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:51,070 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:51,070 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:51,070 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:51,070 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:51,070 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:51,071 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:51,071 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:51,071 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:51,071 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:51,071 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:51,071 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,071 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:51,071 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:51,071 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/ff219c2625db4c1f9e7845bd4f5eb1cd, entries=150, sequenceid=261, filesize=11.8 K 2023-03-26 13:55:51,072 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/33b5fb4caa5743cca9f3bc2040e652c9 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/33b5fb4caa5743cca9f3bc2040e652c9 2023-03-26 13:55:51,073 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:51,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:51,073 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:51,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:51,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:51,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cabcd04[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:51,075 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:51,075 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,076 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/33b5fb4caa5743cca9f3bc2040e652c9, entries=150, sequenceid=261, filesize=11.8 K 2023-03-26 13:55:51,076 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:51,077 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:51,077 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:51,077 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/1da4573419274ad29a3f54dc49152257 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/1da4573419274ad29a3f54dc49152257 2023-03-26 13:55:51,077 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,077 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:51,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:51,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:51,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:51,080 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/1da4573419274ad29a3f54dc49152257, entries=150, sequenceid=261, filesize=11.8 K 2023-03-26 13:55:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,081 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2990228151772e454b24e80da60e1bb2 in 1287ms, sequenceid=261, compaction requested=true 2023-03-26 13:55:51,081 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:51,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:51,081 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:51,081 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:51,081 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:51,082 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:51,082 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:51,082 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:51,082 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:51,082 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:51,082 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:51,082 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:51,083 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,083 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,083 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:51,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,085 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36670 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:51,085 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/A is initiating minor compaction (all files) 2023-03-26 13:55:51,086 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/A in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:51,086 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/49fb33e7f1034d10ab91850b02b55422, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/51a5ffda71ce49339fc158ca8f0f31a1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/ff219c2625db4c1f9e7845bd4f5eb1cd] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=35.8 K 2023-03-26 13:55:51,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,086 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 49fb33e7f1034d10ab91850b02b55422, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1679838947891 2023-03-26 13:55:51,086 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,086 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36670 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:51,086 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/B is initiating minor compaction (all files) 2023-03-26 13:55:51,086 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/B in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:51,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:51,086 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/19be7a99b67c4cb283b2208717809e1b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9a86e8fa9a20488e8152851a2643c1b5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/33b5fb4caa5743cca9f3bc2040e652c9] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=35.8 K 2023-03-26 13:55:51,087 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 51a5ffda71ce49339fc158ca8f0f31a1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1679838948027 2023-03-26 13:55:51,087 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 19be7a99b67c4cb283b2208717809e1b, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1679838947891 2023-03-26 13:55:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,087 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting ff219c2625db4c1f9e7845bd4f5eb1cd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1679838949166 2023-03-26 13:55:51,087 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 9a86e8fa9a20488e8152851a2643c1b5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1679838948027 2023-03-26 13:55:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,088 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 33b5fb4caa5743cca9f3bc2040e652c9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1679838949166 2023-03-26 13:55:51,088 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,095 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:51,096 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:51,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:51,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:51,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:51,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:51,096 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:51,096 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:51,096 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:51,101 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,101 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,101 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,101 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,101 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,101 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,101 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:51,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:51,105 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#A#compaction#490 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:51,113 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#B#compaction#491 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:51,145 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/2c9dd99d59df4fe3973f1494be6d65fc as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/2c9dd99d59df4fe3973f1494be6d65fc 2023-03-26 13:55:51,152 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/A of 2990228151772e454b24e80da60e1bb2 into 2c9dd99d59df4fe3973f1494be6d65fc(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:51,152 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:51,152 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/A, priority=13, startTime=1679838951081; duration=0sec 2023-03-26 13:55:51,152 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:51,152 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:55:51,155 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48604 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:55:51,155 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/C is initiating minor compaction (all files) 2023-03-26 13:55:51,155 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/C in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:51,155 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c788d45f7f4344649f7b01404cc60af1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b04e3dc6d46949b793e1a8a9d37bb931, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/8e04ee371b9a49baba1f4ee2f6299b3d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/1da4573419274ad29a3f54dc49152257] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=47.5 K 2023-03-26 13:55:51,155 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting c788d45f7f4344649f7b01404cc60af1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=207, earliestPutTs=1679838947868 2023-03-26 13:55:51,156 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting b04e3dc6d46949b793e1a8a9d37bb931, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1679838947891 2023-03-26 13:55:51,156 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 8e04ee371b9a49baba1f4ee2f6299b3d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1679838948027 2023-03-26 13:55:51,157 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 1da4573419274ad29a3f54dc49152257, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1679838949166 2023-03-26 13:55:51,168 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#C#compaction#492 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:51,182 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/bb49e162b9ff49c9952bdc1d88a0c854 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/bb49e162b9ff49c9952bdc1d88a0c854 2023-03-26 13:55:51,187 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2990228151772e454b24e80da60e1bb2/C of 2990228151772e454b24e80da60e1bb2 into bb49e162b9ff49c9952bdc1d88a0c854(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:51,188 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:51,188 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/C, priority=12, startTime=1679838951082; duration=0sec 2023-03-26 13:55:51,188 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:51,196 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:51,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:51,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:51,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:51,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:51,204 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:51,204 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:51,204 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:51,204 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:51,204 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,204 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,206 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:51,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:51,206 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:51,208 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,208 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:51,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:51,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,209 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:51,209 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:51,209 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:51,209 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:51,209 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:51,209 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:51,209 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:51,210 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:51,210 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:51,210 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:51,210 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-26 13:55:51,210 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:51,210 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:51,210 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:51,210 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:51,211 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:51,211 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:51,226 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/54f35ccc19e24274acdee1fa62c5fc93 2023-03-26 13:55:51,542 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/7f8097a9f4834835a266a76071d9920e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/7f8097a9f4834835a266a76071d9920e 2023-03-26 13:55:51,545 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/B of 2990228151772e454b24e80da60e1bb2 into 7f8097a9f4834835a266a76071d9920e(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:51,545 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:51,545 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/B, priority=13, startTime=1679838951082; duration=0sec 2023-03-26 13:55:51,546 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:51,644 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/901f631fa9b4467c99cee436875e420c 2023-03-26 13:55:51,653 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/ea32489048714164ac2921f7b942c8cc 2023-03-26 13:55:51,657 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/54f35ccc19e24274acdee1fa62c5fc93 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/54f35ccc19e24274acdee1fa62c5fc93 2023-03-26 13:55:51,661 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/54f35ccc19e24274acdee1fa62c5fc93, entries=150, sequenceid=278, filesize=11.9 K 2023-03-26 13:55:51,662 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/901f631fa9b4467c99cee436875e420c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/901f631fa9b4467c99cee436875e420c 2023-03-26 13:55:51,665 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/901f631fa9b4467c99cee436875e420c, entries=150, sequenceid=278, filesize=11.9 K 2023-03-26 13:55:51,666 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/ea32489048714164ac2921f7b942c8cc as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ea32489048714164ac2921f7b942c8cc 2023-03-26 13:55:51,669 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ea32489048714164ac2921f7b942c8cc, entries=150, sequenceid=278, filesize=11.9 K 2023-03-26 13:55:51,669 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 2990228151772e454b24e80da60e1bb2 in 459ms, sequenceid=278, compaction requested=false 2023-03-26 13:55:51,669 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:51,669 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:51,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:51,672 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,672 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:51,672 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,672 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:51,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:51,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44d4f189[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:51,673 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:51,674 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,674 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,675 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:51,675 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:51,675 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:51,675 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,675 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:51,676 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:51,676 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:51,677 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:51,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:51,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:51,679 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:51,679 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:51,679 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:51,679 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:51,679 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:51,681 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,682 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,682 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:51,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:51,682 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,685 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,688 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,688 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:51,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:51,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:51,688 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:51,688 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,688 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,688 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,688 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,688 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,688 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,688 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:51,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:51,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:51,688 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:51,689 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:51,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:51,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:51,789 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:51,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:51,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:51,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:51,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:51,792 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:51,792 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:51,792 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:51,792 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:51,792 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,792 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,793 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,793 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:51,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:51,794 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,794 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:51,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:51,795 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:51,795 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:51,795 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:51,795 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:51,795 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:51,795 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,795 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:51,796 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:51,798 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,798 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:51,798 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,798 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:51,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32e94acc[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:51,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:51,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,801 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,801 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:51,801 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:51,801 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:51,801 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,801 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:51,802 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,802 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:51,802 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:51,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:51,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:51,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:51,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:51,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:51,806 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:51,806 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:51,806 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:51,806 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:51,806 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:51,807 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,807 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,808 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:51,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:51,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:51,808 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,810 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,813 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:51,813 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:51,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:51,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:51,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:51,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,815 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,814 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:51,815 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,815 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,815 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:51,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:51,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:51,815 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:51,815 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:51,915 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:51,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:51,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:51,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:51,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:51,920 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:51,920 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:51,920 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:51,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:51,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,922 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:51,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:51,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:51,922 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:51,924 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:51,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,925 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:51,925 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:51,925 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:51,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:51,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:51,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:51,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:51,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:51,925 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:51,925 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:51,925 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:51,925 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:51,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:51,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:51,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:51,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:51,927 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,927 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:51,927 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,927 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:51,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:51,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37ad47d2[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:51,930 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:51,930 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,931 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:51,931 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:51,931 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:51,931 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,932 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:51,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,939 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:51,939 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:51,939 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:51,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:51,942 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:51,942 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:51,942 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:51,942 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:51,942 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:51,943 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,943 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:51,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,943 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:51,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:51,946 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,948 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:55:51,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:51,950 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:51,950 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:51,950 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:51,950 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:51,950 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:51,950 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:51,956 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:51,956 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:51,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:51,956 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,956 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:51,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:51,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:51,956 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:51,956 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:51,957 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,957 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,957 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:51,957 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:51,957 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:51,957 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:51,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:51,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:51,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839011977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839011976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:51,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1679838951964, max timestamp=1679838951985Num uniques 859; 2023-03-26 13:55:51,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1679838951964, max timestamp=1679838951985Num uniques 859; 2023-03-26 13:55:51,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1679838951964, max timestamp=1679838951985Num uniques 859; 2023-03-26 13:55:51,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:55:51,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:55:51,987 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:55:52,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:52,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839012013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:52,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839012019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:52,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839012020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,057 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:52,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:52,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:52,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:52,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:52,060 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:52,060 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:52,060 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:52,060 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:52,061 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,061 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,063 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:52,063 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:52,063 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,063 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:52,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:52,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:52,064 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:52,064 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:52,064 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:52,064 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:52,064 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:52,064 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:52,064 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:52,064 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:52,064 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:52,064 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:52,064 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:52,064 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,064 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:52,064 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:52,066 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,066 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,066 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,066 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@558c6fce[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:52,067 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:52,067 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,068 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,068 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:52,069 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:52,069 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:52,069 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,069 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:52,070 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,070 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,070 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:52,070 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:52,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:52,072 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:52,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:52,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:52,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:52,072 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:52,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,075 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,076 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,079 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:52,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:52,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:52,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:52,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:52,079 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:52,079 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:52,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:52,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:52,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:52,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:52,093 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:52,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839012088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:52,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839012089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:52,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839012122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,128 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:52,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839012124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:52,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839012126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,179 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:52,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:52,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:52,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:52,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:52,182 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:52,182 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:52,182 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:52,182 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:52,182 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,182 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,184 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:52,184 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:52,184 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,184 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:52,185 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:52,185 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:52,185 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:52,185 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:52,185 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:52,186 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:52,186 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:52,186 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:52,186 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:52,187 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,188 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,188 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:52,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3637944f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:52,190 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:52,190 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,193 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,193 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:52,193 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:52,193 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:52,193 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,193 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:52,194 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,194 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:52,194 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:52,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:52,197 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:52,197 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:52,197 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:52,197 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:52,198 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:52,199 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,199 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,200 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,200 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,201 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,203 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,203 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:52,203 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:52,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:52,203 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,203 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:52,203 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:52,204 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:52,204 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,204 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,204 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,204 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:52,204 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:52,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:52,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:52,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839012294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:52,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839012296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,304 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:52,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:52,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:52,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:52,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:52,307 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:52,307 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:52,307 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:52,307 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:52,308 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,308 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,309 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:52,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:52,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:52,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:52,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:52,311 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:52,311 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:52,311 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:52,311 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:52,311 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:52,311 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:52,311 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:52,311 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:52,312 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:52,312 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:52,312 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:52,312 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,312 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:52,312 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:52,313 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,313 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,313 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,313 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:52,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:52,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ef0076f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:52,315 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:52,316 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,317 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,317 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:52,317 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:52,317 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:52,317 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,317 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:52,318 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,318 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:52,318 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,318 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:52,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:52,321 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:52,321 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:52,321 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:52,321 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:52,321 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:52,324 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,324 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,324 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,324 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:52,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,325 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,327 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,327 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,327 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,327 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:52,327 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:52,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:52,327 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,328 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:52,328 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:52,328 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:52,327 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:52,328 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:52,328 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:52,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:52,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:52,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:52,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59286 deadline: 1679839012327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:52,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839012330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:52,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839012331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,393 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/b46ec9ef145b4c098a4120c61805d9b1 2023-03-26 13:55:52,428 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:52,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:52,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:52,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:52,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:52,431 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:52,431 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:52,431 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:52,432 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:52,433 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,433 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,436 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:52,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:52,437 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,437 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:52,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:52,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:52,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:52,438 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:52,438 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:52,438 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:52,438 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:52,438 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:52,446 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,446 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,446 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,446 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:52,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@133e4ab4[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:52,447 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:52,447 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,448 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:52,449 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:52,449 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,449 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:52,450 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,450 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:52,450 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,450 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:52,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:52,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:52,453 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:52,453 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:52,453 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:52,453 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:52,453 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:52,454 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,454 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,454 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:52,455 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,456 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,459 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,459 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,459 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,459 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,459 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,459 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,459 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:52,459 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:52,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:52,459 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:52,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:52,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:52,459 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:52,460 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:52,460 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:52,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:52,560 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:52,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:52,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:52,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:52,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:52,566 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:52,566 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:52,566 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:52,566 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:52,566 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,566 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,567 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,568 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:52,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:52,568 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,568 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:52,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:52,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:52,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:52,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:52,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:52,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:52,569 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:52,569 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:52,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:52,570 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:52,570 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:52,570 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:52,570 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:52,570 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,570 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:52,570 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:52,571 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,572 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,572 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:52,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@257aa502[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:52,573 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:52,573 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,575 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,575 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:52,575 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:52,575 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:52,575 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,575 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:52,577 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,577 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,577 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:52,577 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:52,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:52,581 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:52,581 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:52,581 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:52,581 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:52,581 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:52,582 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,582 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,582 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:52,583 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:52,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,584 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:52,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:52,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:52,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:52,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,587 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:52,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:52,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:52,588 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:52,588 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:52,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:52,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:52,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:52,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:52,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839012601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:52,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839012602, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:52,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839012632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:52,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839012636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:52,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839012636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,688 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:52,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:52,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:52,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:52,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:52,696 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:52,696 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:52,696 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:52,696 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:52,696 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,696 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,698 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:52,698 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,698 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,698 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:52,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:52,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:52,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:52,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:52,700 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:52,700 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:52,700 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:52,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:52,704 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:52,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:52,704 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:52,704 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:52,704 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:52,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:52,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:52,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:52,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:52,710 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,710 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,710 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,710 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:52,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:52,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f65c80e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:52,712 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:52,712 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,715 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:52,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:52,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:52,716 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,716 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:52,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,717 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:52,717 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:52,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,717 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:52,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:52,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:52,720 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:52,720 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:52,720 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:52,720 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:52,720 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:52,723 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,723 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,723 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,725 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:52,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:52,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,726 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,730 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:52,730 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,730 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:52,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:52,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:52,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:52,730 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:52,730 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,730 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:52,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:52,730 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:52,730 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,730 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,730 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:52,730 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,730 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,821 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/a2abeaf8a96c4416a1ae558b23766d0d 2023-03-26 13:55:52,830 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:52,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:52,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:52,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:52,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:52,833 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:52,834 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:52,834 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:52,834 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:52,836 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,836 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,837 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:52,837 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:52,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:52,838 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,838 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:52,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,838 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:52,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:52,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:52,839 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:52,839 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:52,839 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:52,839 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:52,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:52,841 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,841 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,841 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,841 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:52,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@616079f2[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:52,842 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:52,842 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,846 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,846 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:52,846 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:52,846 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:52,846 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,846 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:52,847 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,848 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:52,848 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,848 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:52,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:52,850 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:52,850 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:52,850 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:52,850 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:52,850 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:52,852 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,852 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,852 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:52,852 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,853 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:52,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:52,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:52,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:52,856 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:52,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:52,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:52,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:52,857 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:52,857 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:52,957 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:52,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:52,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:52,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:52,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:52,960 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:52,960 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:52,960 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:52,960 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:52,960 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,961 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,963 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,963 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:52,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:52,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:52,963 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,963 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:52,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:52,964 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:52,965 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:52,965 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:52,965 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:52,965 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:52,965 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:52,965 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:52,967 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,967 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,967 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,967 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:52,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@eae480[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:52,968 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:52,968 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,969 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,969 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:52,970 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:52,970 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:52,970 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,970 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:52,971 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,971 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:52,971 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,971 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:52,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:52,973 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:52,973 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:52,973 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:52,973 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:52,973 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:52,976 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,977 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:52,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,977 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:52,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:52,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,979 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:52,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,980 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:52,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:52,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,982 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,982 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,982 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:52,982 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:52,982 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:52,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:52,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:52,982 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:52,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:52,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:52,982 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:52,982 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:52,982 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:52,982 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,982 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:52,982 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:52,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:52,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:52,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:53,082 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:53,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:53,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:53,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:53,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:53,085 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:53,086 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:53,086 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:53,086 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:53,086 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,086 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,087 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:53,087 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:53,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:53,088 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,088 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:53,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:53,089 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:53,089 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:53,089 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:53,089 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:53,089 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:53,089 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,089 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:53,090 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:53,091 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,091 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:53,091 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:53,091 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:53,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:53,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70c8032a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:53,094 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:53,094 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,096 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,096 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:53,096 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:53,096 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:53,096 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,096 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:53,097 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,097 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:53,098 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:53,098 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:53,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:53,100 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:53,100 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:53,100 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:53,100 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:53,100 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:53,101 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:53,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:53,102 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,104 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,104 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:53,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,106 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:53,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,108 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,108 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,108 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:53,108 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:53,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:53,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:53,108 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:53,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:53,108 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:53,108 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:53,108 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:53,108 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,108 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,108 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,108 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:53,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:53,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:53,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59340 deadline: 1679839013109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:53,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59354 deadline: 1679839013109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:53,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839013137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:53,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839013160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:53,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59310 deadline: 1679839013160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,208 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:53,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:53,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:53,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:53,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:53,211 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:53,211 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:53,211 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:53,211 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:53,212 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,212 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:53,214 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:53,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:53,214 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,214 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:53,215 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:53,215 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:53,216 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:53,216 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:53,216 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:53,216 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:53,216 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:53,216 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:53,216 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:53,216 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:53,216 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:53,216 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,216 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:53,216 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:53,217 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,217 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:53,218 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,218 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:53,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:53,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71ec9085[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:53,219 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:53,219 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,220 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,221 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:53,221 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:53,221 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:53,221 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,221 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:53,226 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:53,226 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:53,226 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:53,226 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:53,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:53,228 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:53,228 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:53,228 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:53,228 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:53,228 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:53,232 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,232 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:53,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:53,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:53,232 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,233 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,233 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/c4d6ab73383644019e1fac8d42ab0aa7 2023-03-26 13:55:53,233 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:53,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,236 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,236 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,236 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,236 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:53,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:53,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:53,236 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:53,236 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:53,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:53,236 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,236 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:53,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:53,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:53,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:53,236 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:53,236 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,237 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,237 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:53,237 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/b46ec9ef145b4c098a4120c61805d9b1 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b46ec9ef145b4c098a4120c61805d9b1 2023-03-26 13:55:53,242 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b46ec9ef145b4c098a4120c61805d9b1, entries=150, sequenceid=292, filesize=11.9 K 2023-03-26 13:55:53,242 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/a2abeaf8a96c4416a1ae558b23766d0d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a2abeaf8a96c4416a1ae558b23766d0d 2023-03-26 13:55:53,245 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a2abeaf8a96c4416a1ae558b23766d0d, entries=100, sequenceid=292, filesize=9.5 K 2023-03-26 13:55:53,246 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/c4d6ab73383644019e1fac8d42ab0aa7 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c4d6ab73383644019e1fac8d42ab0aa7 2023-03-26 13:55:53,249 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c4d6ab73383644019e1fac8d42ab0aa7, entries=100, sequenceid=292, filesize=9.5 K 2023-03-26 13:55:53,249 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2990228151772e454b24e80da60e1bb2 in 1301ms, sequenceid=292, compaction requested=true 2023-03-26 13:55:53,249 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:53,250 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:53,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:53,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:53,250 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:53,250 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:53,250 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37068 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:53,250 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34628 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:53,250 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/A is initiating minor compaction (all files) 2023-03-26 13:55:53,251 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/B is initiating minor compaction (all files) 2023-03-26 13:55:53,251 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/A in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:53,251 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/B in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:53,251 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/2c9dd99d59df4fe3973f1494be6d65fc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/54f35ccc19e24274acdee1fa62c5fc93, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b46ec9ef145b4c098a4120c61805d9b1] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=36.2 K 2023-03-26 13:55:53,251 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/7f8097a9f4834835a266a76071d9920e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/901f631fa9b4467c99cee436875e420c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a2abeaf8a96c4416a1ae558b23766d0d] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=33.8 K 2023-03-26 13:55:53,251 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 2c9dd99d59df4fe3973f1494be6d65fc, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1679838949166 2023-03-26 13:55:53,251 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 7f8097a9f4834835a266a76071d9920e, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1679838949166 2023-03-26 13:55:53,251 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 54f35ccc19e24274acdee1fa62c5fc93, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1679838949801 2023-03-26 13:55:53,251 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 901f631fa9b4467c99cee436875e420c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1679838949801 2023-03-26 13:55:53,252 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting b46ec9ef145b4c098a4120c61805d9b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1679838951940 2023-03-26 13:55:53,252 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting a2abeaf8a96c4416a1ae558b23766d0d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1679838951947 2023-03-26 13:55:53,257 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#B#compaction#499 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:53,258 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#A#compaction#500 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:53,337 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:53,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:53,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:53,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:53,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:53,340 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:53,340 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:53,340 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:53,340 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:53,340 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,340 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,342 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:53,342 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:53,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,344 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,344 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:53,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,344 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:53,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:53,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:53,345 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:53,345 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:53,345 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:53,345 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-26 13:55:53,345 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:53,346 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:53,346 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:53,346 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:53,346 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:53,346 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:53,354 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/040ce1b4fc94475491a6c15902639ac9 2023-03-26 13:55:53,362 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/1d44e350151d4641bbe8815c96f927c7 2023-03-26 13:55:53,666 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/9e1ef49fa8424a95818a5d8708bf723d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9e1ef49fa8424a95818a5d8708bf723d 2023-03-26 13:55:53,672 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/B of 2990228151772e454b24e80da60e1bb2 into 9e1ef49fa8424a95818a5d8708bf723d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:53,672 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:53,672 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/B, priority=13, startTime=1679838953250; duration=0sec 2023-03-26 13:55:53,672 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:53,673 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:53,673 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34560 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:53,674 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/C is initiating minor compaction (all files) 2023-03-26 13:55:53,674 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/C in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:53,674 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/bb49e162b9ff49c9952bdc1d88a0c854, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ea32489048714164ac2921f7b942c8cc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c4d6ab73383644019e1fac8d42ab0aa7] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=33.8 K 2023-03-26 13:55:53,674 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting bb49e162b9ff49c9952bdc1d88a0c854, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1679838949166 2023-03-26 13:55:53,675 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting ea32489048714164ac2921f7b942c8cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1679838949801 2023-03-26 13:55:53,675 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting c4d6ab73383644019e1fac8d42ab0aa7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1679838951947 2023-03-26 13:55:53,676 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/cb0edf2d5ed9489e8cda52dac1d69c9d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/cb0edf2d5ed9489e8cda52dac1d69c9d 2023-03-26 13:55:53,680 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/A of 2990228151772e454b24e80da60e1bb2 into cb0edf2d5ed9489e8cda52dac1d69c9d(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:53,681 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:53,681 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/A, priority=13, startTime=1679838953249; duration=0sec 2023-03-26 13:55:53,681 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#C#compaction#504 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:53,681 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:53,691 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/a7980d2aad564ea783316c95329670d1 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/a7980d2aad564ea783316c95329670d1 2023-03-26 13:55:53,694 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/C of 2990228151772e454b24e80da60e1bb2 into a7980d2aad564ea783316c95329670d1(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:53,694 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:53,694 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/C, priority=13, startTime=1679838953250; duration=0sec 2023-03-26 13:55:53,694 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:53,770 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/130c02e135f44720b673ab5372d5463e 2023-03-26 13:55:53,774 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/040ce1b4fc94475491a6c15902639ac9 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/040ce1b4fc94475491a6c15902639ac9 2023-03-26 13:55:53,777 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/040ce1b4fc94475491a6c15902639ac9, entries=150, sequenceid=326, filesize=11.9 K 2023-03-26 13:55:53,778 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/1d44e350151d4641bbe8815c96f927c7 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/1d44e350151d4641bbe8815c96f927c7 2023-03-26 13:55:53,782 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/1d44e350151d4641bbe8815c96f927c7, entries=150, sequenceid=326, filesize=11.9 K 2023-03-26 13:55:53,783 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/130c02e135f44720b673ab5372d5463e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/130c02e135f44720b673ab5372d5463e 2023-03-26 13:55:53,785 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/130c02e135f44720b673ab5372d5463e, entries=150, sequenceid=326, filesize=11.9 K 2023-03-26 13:55:53,786 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=0 B/0 for 2990228151772e454b24e80da60e1bb2 in 441ms, sequenceid=326, compaction requested=false 2023-03-26 13:55:53,786 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:53,786 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:53,786 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:53,786 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:53,786 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,786 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:53,786 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:53,789 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,789 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:53,790 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,790 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:53,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:53,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13a1d8a3[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:53,791 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:53,791 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,792 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,792 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:53,793 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,793 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:53,793 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:53,794 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,794 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:53,794 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:53,794 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:53,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:53,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:53,797 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:53,797 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:53,797 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:53,797 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:53,797 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:53,798 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,798 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,798 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:53,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,799 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:53,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:53,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,803 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,803 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,803 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,803 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:53,803 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:53,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:53,803 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:53,803 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:53,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:53,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:53,804 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,804 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,804 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,804 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:53,804 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:53,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:53,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:53,904 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:53,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:53,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:53,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:53,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:53,907 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:53,907 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:53,907 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:53,907 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:53,907 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,907 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,909 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:53,909 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:53,909 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,909 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:53,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:53,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:53,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:53,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:53,910 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:53,910 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:53,910 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:53,910 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:53,910 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:53,911 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:53,911 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:53,911 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:53,911 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:53,911 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:53,911 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:53,911 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,911 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:53,911 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:53,912 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:53,913 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:53,913 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,913 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:53,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:53,914 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:53,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4438e0b3[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:53,914 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,916 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,916 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:53,916 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:53,916 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:53,916 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,916 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:53,921 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,921 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:53,921 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:53,921 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:53,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:53,923 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:53,923 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:53,923 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:53,923 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:53,923 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:53,925 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,925 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,925 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:53,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:53,927 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:53,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,928 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:53,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:53,931 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:53,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:53,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:53,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:53,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:53,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:53,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:53,931 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:53,931 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:53,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:53,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:53,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:53,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:53,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:54,031 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:54,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:54,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:54,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:54,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:54,034 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:54,034 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:54,034 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:54,034 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:54,035 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,035 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,036 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:54,036 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:54,036 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,036 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:54,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:54,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,037 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:54,037 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:54,037 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:54,037 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:54,037 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:54,037 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:54,038 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:54,038 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:54,038 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:54,038 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:54,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:54,041 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,041 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,041 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,041 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:54,043 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74d97a4d[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:54,043 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:54,043 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,048 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,055 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:54,055 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:54,055 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:54,056 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,056 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:54,056 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,056 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,057 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:54,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:54,059 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:54,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:54,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:54,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:54,059 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:54,061 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,061 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,061 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,062 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,063 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,068 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:54,069 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:54,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:54,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:54,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:54,070 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,070 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,070 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:54,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:54,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:54,070 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,070 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,070 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,070 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,070 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,070 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:54,070 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:54,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:54,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:54,152 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:55:54,152 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:54,152 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:54,152 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:54,152 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:54,152 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:54,153 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:54,170 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:54,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:54,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:54,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:54,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:54,184 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:54,184 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:54,184 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:54,184 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:54,185 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,185 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,187 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:54,187 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:54,188 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,188 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:54,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:54,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:54,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:54,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:54,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:54,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:54,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:54,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:54,196 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:54,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:54,197 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:54,197 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:54,197 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:54,197 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:54,197 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:54,197 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,197 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:54,197 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:54,199 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,199 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838954177, max timestamp=1679838954199Num uniques 859; 2023-03-26 13:55:54,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:55:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838954177, max timestamp=1679838954199Num uniques 859; 2023-03-26 13:55:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838954177, max timestamp=1679838954199Num uniques 859; 2023-03-26 13:55:54,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:55:54,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:55:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:54,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61a4b4eb[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:54,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:54,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,203 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,203 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:54,203 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:54,203 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,203 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:54,203 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:54,206 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,206 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:54,206 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,206 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:54,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:54,209 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:54,209 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:54,209 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:54,209 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:54,209 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:54,210 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,210 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,210 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,210 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,212 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:54,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:54,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:54,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:54,214 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:54,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:54,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:54,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,214 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:54,214 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:54,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:54,221 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839014218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839014219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,224 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839014219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839014220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,227 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839014221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,314 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:54,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:54,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:54,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:54,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:54,321 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:54,321 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:54,321 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:54,321 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:54,322 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,322 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:54,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:54,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:54,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:54,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,324 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:54,324 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:54,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:54,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:54,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:54,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:54,325 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,325 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:54,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:54,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:54,325 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:54,325 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:54,325 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:54,325 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:54,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:54,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:54,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:54,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:54,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839014322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,329 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,330 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,330 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:54,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c5d0cbc[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:54,331 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:54,331 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839014328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839014328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,332 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839014328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,332 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:54,332 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,332 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:54,333 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:54,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839014329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,333 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:54,334 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:54,334 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:54,334 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,334 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:54,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:54,336 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:54,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:54,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:54,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:54,337 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:54,338 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,338 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,338 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,339 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,340 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,346 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,346 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,346 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:54,346 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:54,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:54,346 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,346 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:54,347 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,347 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:54,347 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:54,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:54,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:54,346 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:54,347 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:54,347 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,347 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:54,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:54,447 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:54,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:54,449 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:54,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:54,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:54,450 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:54,450 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:54,450 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:54,450 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:54,450 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,450 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,452 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,452 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:54,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:54,452 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,452 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,453 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:54,453 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:54,453 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:54,453 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:54,453 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:54,453 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:54,453 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:54,453 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:54,453 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:54,454 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:54,454 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:54,454 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:54,454 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:54,454 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:54,454 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,454 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:54,454 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:54,455 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,455 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,456 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,456 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:54,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f6346fc[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:54,458 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:54,458 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,459 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,459 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:54,459 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:54,459 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:54,459 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,459 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:54,462 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,462 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:54,462 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:54,462 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:54,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:54,464 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:54,464 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:54,464 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:54,464 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:54,464 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:54,465 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,465 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,465 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:54,467 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,468 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,476 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,476 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:54,476 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:54,476 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:54,476 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,476 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,476 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,476 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,476 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:54,476 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:54,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:54,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:54,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:54,477 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:54,477 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:54,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:54,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:54,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839014530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:59284 deadline: 1679839014533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839014534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839014534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839014534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,577 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:54,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:54,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:54,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:54,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:54,580 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:54,580 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:54,580 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:54,580 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:54,580 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,580 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,581 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:54,581 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:54,582 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,582 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:54,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:54,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:54,583 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:54,583 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:54,583 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:54,583 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:54,583 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:54,583 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:54,583 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:54,583 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:54,583 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:54,583 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:54,583 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:54,583 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,583 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:54,583 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:54,585 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,585 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,585 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,585 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:54,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:54,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4057504f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:54,586 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:54,586 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,587 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,587 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:54,587 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:54,587 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:54,587 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,587 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:54,589 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,589 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:54,589 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,589 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:54,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:54,591 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:54,591 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:54,591 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:54,591 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:54,591 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:54,593 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,593 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,593 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:54,593 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,596 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,597 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,597 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,597 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:54,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:54,597 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:54,597 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,597 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:54,598 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,598 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,598 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,598 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:54,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:54,598 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:54,598 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/29744925d4554a3fa1293980bccf49c5 2023-03-26 13:55:54,598 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:54,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:54,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:54,607 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/33478942a7414c39a3330d403c0911de 2023-03-26 13:55:54,619 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/0c04ff44194a4ab1bbca7cb6c9f092c2 2023-03-26 13:55:54,626 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/29744925d4554a3fa1293980bccf49c5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/29744925d4554a3fa1293980bccf49c5 2023-03-26 13:55:54,629 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/29744925d4554a3fa1293980bccf49c5, entries=100, sequenceid=341, filesize=9.5 K 2023-03-26 13:55:54,629 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/33478942a7414c39a3330d403c0911de as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/33478942a7414c39a3330d403c0911de 2023-03-26 13:55:54,633 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/33478942a7414c39a3330d403c0911de, entries=100, sequenceid=341, filesize=9.5 K 2023-03-26 13:55:54,634 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/0c04ff44194a4ab1bbca7cb6c9f092c2 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/0c04ff44194a4ab1bbca7cb6c9f092c2 2023-03-26 13:55:54,637 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/0c04ff44194a4ab1bbca7cb6c9f092c2, entries=100, sequenceid=341, filesize=9.5 K 2023-03-26 13:55:54,638 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2990228151772e454b24e80da60e1bb2 in 486ms, sequenceid=341, compaction requested=true 2023-03-26 13:55:54,638 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:54,638 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:54,638 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:54,638 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:54,638 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:54,638 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:54,639 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34830 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:54,639 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/B is initiating minor compaction (all files) 2023-03-26 13:55:54,639 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/B in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:54,639 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9e1ef49fa8424a95818a5d8708bf723d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/1d44e350151d4641bbe8815c96f927c7, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/33478942a7414c39a3330d403c0911de] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=34.0 K 2023-03-26 13:55:54,639 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34830 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:54,639 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/A is initiating minor compaction (all files) 2023-03-26 13:55:54,639 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/A in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:54,640 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/cb0edf2d5ed9489e8cda52dac1d69c9d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/040ce1b4fc94475491a6c15902639ac9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/29744925d4554a3fa1293980bccf49c5] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=34.0 K 2023-03-26 13:55:54,640 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 9e1ef49fa8424a95818a5d8708bf723d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1679838949801 2023-03-26 13:55:54,641 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting cb0edf2d5ed9489e8cda52dac1d69c9d, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1679838949801 2023-03-26 13:55:54,641 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 1d44e350151d4641bbe8815c96f927c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1679838952011 2023-03-26 13:55:54,641 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 040ce1b4fc94475491a6c15902639ac9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1679838952011 2023-03-26 13:55:54,641 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 33478942a7414c39a3330d403c0911de, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1679838954118 2023-03-26 13:55:54,642 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 29744925d4554a3fa1293980bccf49c5, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1679838954118 2023-03-26 13:55:54,650 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#B#compaction#508 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:54,651 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#A#compaction#509 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:54,698 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:54,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:54,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:54,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:54,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:54,701 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:54,701 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:54,701 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:54,701 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:54,702 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,702 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,703 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:54,703 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:54,703 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,703 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:54,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:54,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:54,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:54,705 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:54,705 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:54,705 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:54,705 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:54,705 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-26 13:55:54,705 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:54,705 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:54,706 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:54,706 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:54,706 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:54,706 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:54,712 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/25baed7286ca4185b2bdb478d063f3f4 2023-03-26 13:55:54,722 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/7a255090fee943379934d749a4b46316 2023-03-26 13:55:54,729 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=376 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/b73231c61e38487388a9437c3706a52c 2023-03-26 13:55:54,733 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/25baed7286ca4185b2bdb478d063f3f4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/25baed7286ca4185b2bdb478d063f3f4 2023-03-26 13:55:54,736 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/25baed7286ca4185b2bdb478d063f3f4, entries=150, sequenceid=376, filesize=11.9 K 2023-03-26 13:55:54,738 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/7a255090fee943379934d749a4b46316 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/7a255090fee943379934d749a4b46316 2023-03-26 13:55:54,741 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/7a255090fee943379934d749a4b46316, entries=150, sequenceid=376, filesize=11.9 K 2023-03-26 13:55:54,742 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/b73231c61e38487388a9437c3706a52c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b73231c61e38487388a9437c3706a52c 2023-03-26 13:55:54,744 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b73231c61e38487388a9437c3706a52c, entries=150, sequenceid=376, filesize=11.9 K 2023-03-26 13:55:54,744 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-03-26 13:55:54,745 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=0 B/0 for 2990228151772e454b24e80da60e1bb2 in 40ms, sequenceid=376, compaction requested=true 2023-03-26 13:55:54,745 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:54,745 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:54,745 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:54,748 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:54,750 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,750 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,751 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,751 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:54,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@490b742[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:54,754 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:54,754 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,756 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,756 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:54,757 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:54,757 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:54,757 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,757 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:54,758 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,758 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:54,758 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,758 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:54,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:54,760 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:54,760 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:54,760 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:54,760 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:54,760 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:54,762 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,762 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,762 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:54,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,764 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,765 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,770 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:54,770 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:54,770 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:54,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:54,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:54,770 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,770 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:54,771 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:54,771 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:54,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:54,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:54,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:54,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:54,846 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:55:54,846 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:54,847 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:54,847 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:54,847 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:54,847 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:54,847 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:54,853 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/52464a85a7c240a885959214c72a8364 2023-03-26 13:55:54,871 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:54,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:54,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838954856, max timestamp=1679838954871Num uniques 859; 2023-03-26 13:55:54,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838954856, max timestamp=1679838954871Num uniques 859; 2023-03-26 13:55:54,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:55:54,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838954856, max timestamp=1679838954871Num uniques 859; 2023-03-26 13:55:54,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:55:54,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:55:54,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:54,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:54,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:54,874 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:54,874 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:54,875 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:54,875 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:54,875 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,875 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,876 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,876 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:54,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:54,876 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,876 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,881 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:54,881 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:54,881 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:54,882 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:54,882 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:54,882 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:54,882 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:54,882 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:54,884 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,884 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,884 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,884 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:54,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@103c7380[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:54,886 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:54,886 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,887 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:54,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:54,887 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:54,888 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,888 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:54,890 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:54,890 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,890 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,890 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:54,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:54,892 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:54,892 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:54,892 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:54,892 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:54,892 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:54,893 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,893 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,893 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:54,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:54,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:54,894 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,895 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,897 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,897 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,897 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:54,897 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:54,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:54,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:54,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:54,897 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:54,897 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:54,897 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,897 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,897 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:54,897 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:54,897 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:54,897 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:54,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:54,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:54,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839014899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,912 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839014900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839014901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839014908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:54,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839014909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:54,997 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:54,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:55,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:55,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:55,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:55,001 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:55,001 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:55,001 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:55,001 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:55,001 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,001 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,006 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:55,006 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:55,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:55,006 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,007 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:55,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:55,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:55,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:55,008 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,008 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:55,008 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:55,008 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:55,008 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:55,010 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,010 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,010 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,010 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:55,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@750d83dd[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:55,011 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:55,011 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,012 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,013 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:55,013 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:55,013 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:55,013 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,013 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:55,014 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,014 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:55,014 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,014 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839015010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:55,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:55,017 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:55,017 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:55,017 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:55,017 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:55,017 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:55,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839015013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839015013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839015017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839015017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,025 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,025 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,025 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:55,026 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:55,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:55,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,027 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:55,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:55,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:55,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,029 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:55,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:55,029 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:55,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:55,029 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:55,029 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:55,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:55,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:55,066 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/3490784ced664c1cad277fe3a50694ae as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/3490784ced664c1cad277fe3a50694ae 2023-03-26 13:55:55,069 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/8b4586b602584320a8af8e3b0a988e98 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/8b4586b602584320a8af8e3b0a988e98 2023-03-26 13:55:55,070 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/A of 2990228151772e454b24e80da60e1bb2 into 3490784ced664c1cad277fe3a50694ae(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:55,070 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:55,070 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/A, priority=13, startTime=1679838954638; duration=0sec 2023-03-26 13:55:55,071 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:55,071 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:55:55,072 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46948 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:55:55,072 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/C is initiating minor compaction (all files) 2023-03-26 13:55:55,072 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/C in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,072 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/a7980d2aad564ea783316c95329670d1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/130c02e135f44720b673ab5372d5463e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/0c04ff44194a4ab1bbca7cb6c9f092c2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b73231c61e38487388a9437c3706a52c] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=45.8 K 2023-03-26 13:55:55,072 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting a7980d2aad564ea783316c95329670d1, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1679838949801 2023-03-26 13:55:55,073 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 130c02e135f44720b673ab5372d5463e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1679838952011 2023-03-26 13:55:55,073 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 0c04ff44194a4ab1bbca7cb6c9f092c2, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1679838954118 2023-03-26 13:55:55,073 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/B of 2990228151772e454b24e80da60e1bb2 into 8b4586b602584320a8af8e3b0a988e98(size=12.7 K), total size for store is 24.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:55,073 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting b73231c61e38487388a9437c3706a52c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1679838954206 2023-03-26 13:55:55,073 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:55,073 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/B, priority=13, startTime=1679838954638; duration=0sec 2023-03-26 13:55:55,074 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:55,080 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#C#compaction#515 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:55,129 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:55,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:55,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:55,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:55,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:55,133 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:55,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:55,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:55,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:55,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,134 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:55,134 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:55,135 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,135 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:55,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:55,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:55,136 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,136 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:55,136 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:55,136 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:55,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:55,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:55,138 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,138 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,138 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,138 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:55,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f22701b[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:55,140 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:55,140 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,142 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,142 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:55,142 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:55,142 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:55,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:55,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:55,143 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:55,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,143 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:55,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:55,145 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:55,145 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:55,145 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:55,145 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:55,146 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:55,147 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,147 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,147 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,148 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,149 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,151 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,151 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:55,151 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:55,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:55,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:55,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:55,151 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:55,151 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:55,152 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:55,152 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:55,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:55,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:55,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839015219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839015219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839015220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839015225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839015225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,252 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:55,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:55,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:55,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:55,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:55,255 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:55,255 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:55,255 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:55,256 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:55,260 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/a8208a538dcf4e54be4ef576f8de31e9 2023-03-26 13:55:55,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,263 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:55,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:55,263 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:55,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:55,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:55,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:55,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:55,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:55,265 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:55,268 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,268 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:55,268 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:55,268 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:55,268 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:55,270 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,270 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,270 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,270 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:55,272 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e09ec1a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:55,272 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:55,272 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,273 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:55,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:55,273 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,273 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:55,273 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:55,274 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:55,274 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:55,274 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,274 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:55,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:55,278 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:55,278 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:55,278 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:55,278 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:55,278 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:55,282 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/8aade3946dd04b8b8605e7f23dce51d0 2023-03-26 13:55:55,285 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,285 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:55,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:55,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,286 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,287 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/52464a85a7c240a885959214c72a8364 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/52464a85a7c240a885959214c72a8364 2023-03-26 13:55:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,288 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,290 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,293 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/52464a85a7c240a885959214c72a8364, entries=200, sequenceid=386, filesize=14.3 K 2023-03-26 13:55:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,294 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,294 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,294 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,294 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:55,294 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:55,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:55,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:55,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:55,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:55,294 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,294 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:55,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:55,295 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:55,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:55,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,295 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:55,301 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/a8208a538dcf4e54be4ef576f8de31e9 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a8208a538dcf4e54be4ef576f8de31e9 2023-03-26 13:55:55,305 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a8208a538dcf4e54be4ef576f8de31e9, entries=150, sequenceid=386, filesize=11.9 K 2023-03-26 13:55:55,305 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/8aade3946dd04b8b8605e7f23dce51d0 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/8aade3946dd04b8b8605e7f23dce51d0 2023-03-26 13:55:55,309 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/8aade3946dd04b8b8605e7f23dce51d0, entries=150, sequenceid=386, filesize=11.9 K 2023-03-26 13:55:55,310 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2990228151772e454b24e80da60e1bb2 in 464ms, sequenceid=386, compaction requested=true 2023-03-26 13:55:55,310 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:55,310 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:55,310 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:55,310 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-26 13:55:55,310 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:55,311 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39816 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:55,311 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/A is initiating minor compaction (all files) 2023-03-26 13:55:55,311 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/A in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,311 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/3490784ced664c1cad277fe3a50694ae, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/25baed7286ca4185b2bdb478d063f3f4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/52464a85a7c240a885959214c72a8364] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=38.9 K 2023-03-26 13:55:55,311 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 3490784ced664c1cad277fe3a50694ae, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1679838952011 2023-03-26 13:55:55,312 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 25baed7286ca4185b2bdb478d063f3f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1679838954206 2023-03-26 13:55:55,312 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 52464a85a7c240a885959214c72a8364, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1679838954840 2023-03-26 13:55:55,317 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#A#compaction#517 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:55,331 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/4cbc3880fba546b3845e6c8756d6fc27 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/4cbc3880fba546b3845e6c8756d6fc27 2023-03-26 13:55:55,335 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/A of 2990228151772e454b24e80da60e1bb2 into 4cbc3880fba546b3845e6c8756d6fc27(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:55,335 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:55,335 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/A, priority=13, startTime=1679838955310; duration=0sec 2023-03-26 13:55:55,335 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-26 13:55:55,335 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 4 compacting, 1 eligible, 16 blocking 2023-03-26 13:55:55,335 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 13:55:55,335 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 13:55:55,335 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. because compaction request was cancelled 2023-03-26 13:55:55,336 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:55,336 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:55,336 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/B is initiating minor compaction (all files) 2023-03-26 13:55:55,336 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/B in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,337 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/8b4586b602584320a8af8e3b0a988e98, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/7a255090fee943379934d749a4b46316, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a8208a538dcf4e54be4ef576f8de31e9] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=36.5 K 2023-03-26 13:55:55,337 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 8b4586b602584320a8af8e3b0a988e98, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1679838952011 2023-03-26 13:55:55,338 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 7a255090fee943379934d749a4b46316, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1679838954206 2023-03-26 13:55:55,338 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting a8208a538dcf4e54be4ef576f8de31e9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1679838954840 2023-03-26 13:55:55,344 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#B#compaction#518 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:55,351 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/a720ae392ac14e519394b64413549435 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a720ae392ac14e519394b64413549435 2023-03-26 13:55:55,355 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/B of 2990228151772e454b24e80da60e1bb2 into a720ae392ac14e519394b64413549435(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:55,355 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:55,355 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/B, priority=13, startTime=1679838955310; duration=0sec 2023-03-26 13:55:55,355 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:55,395 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:55,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:55,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:55,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:55,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:55,398 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:55,398 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:55,398 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:55,398 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:55,399 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,399 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,400 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:55,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:55,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,400 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:55,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:55,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:55,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:55,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:55,401 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:55,401 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:55,401 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:55,401 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:55,402 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:55,402 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,402 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:55,402 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-26 13:55:55,402 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:55,402 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:55,402 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:55,402 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:55,402 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:55,402 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:55,413 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/f05690db23ef4f7c9c9999096747a45b 2023-03-26 13:55:55,421 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/f8860e3621684134a40aaa8ecdb3fea0 2023-03-26 13:55:55,440 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/5def744979d94ed5bf446bdb4001d395 2023-03-26 13:55:55,446 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/f05690db23ef4f7c9c9999096747a45b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/f05690db23ef4f7c9c9999096747a45b 2023-03-26 13:55:55,450 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/f05690db23ef4f7c9c9999096747a45b, entries=150, sequenceid=426, filesize=11.9 K 2023-03-26 13:55:55,450 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/f8860e3621684134a40aaa8ecdb3fea0 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/f8860e3621684134a40aaa8ecdb3fea0 2023-03-26 13:55:55,453 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/f8860e3621684134a40aaa8ecdb3fea0, entries=150, sequenceid=426, filesize=11.9 K 2023-03-26 13:55:55,454 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/5def744979d94ed5bf446bdb4001d395 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/5def744979d94ed5bf446bdb4001d395 2023-03-26 13:55:55,459 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/5def744979d94ed5bf446bdb4001d395, entries=150, sequenceid=426, filesize=11.9 K 2023-03-26 13:55:55,459 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for 2990228151772e454b24e80da60e1bb2 in 57ms, sequenceid=426, compaction requested=false 2023-03-26 13:55:55,459 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:55,460 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,460 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:55,460 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:55,460 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,460 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:55,460 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,465 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,465 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:55,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:55,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d5a2e28[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:55,467 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:55,467 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,468 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,468 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:55,468 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:55,468 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:55,468 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,468 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:55,470 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:55,470 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:55,470 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,470 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:55,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:55,472 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:55,472 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:55,472 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:55,472 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:55,472 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:55,475 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,475 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,475 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,476 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,477 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:55,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:55,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:55,480 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:55,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,480 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:55,480 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:55,481 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:55,481 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:55,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:55,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:55,487 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/9d8ae4309eb64ee9adc328baa44c7ee7 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/9d8ae4309eb64ee9adc328baa44c7ee7 2023-03-26 13:55:55,490 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2990228151772e454b24e80da60e1bb2/C of 2990228151772e454b24e80da60e1bb2 into 9d8ae4309eb64ee9adc328baa44c7ee7(size=12.7 K), total size for store is 36.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:55,490 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:55,490 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/C, priority=12, startTime=1679838954638; duration=0sec 2023-03-26 13:55:55,491 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:55,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:55,533 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:55:55,533 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:55,533 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:55,533 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:55,533 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:55,533 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:55,533 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:55,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839015555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16434, totalHeapSize=37576, min timestamp=1679838955540, max timestamp=1679838955556Num uniques 859; 2023-03-26 13:55:55,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679838955549, max timestamp=1679838955556Num uniques 859; 2023-03-26 13:55:55,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=16434, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-26 13:55:55,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16434, totalHeapSize=37576, min timestamp=1679838955540, max timestamp=1679838955556Num uniques 859; 2023-03-26 13:55:55,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:55:55,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=16434, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-26 13:55:55,580 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:55,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:55,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:55,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:55,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:55,583 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:55,583 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:55,583 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:55,583 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:55,583 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,584 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,585 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:55,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:55,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,585 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:55,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:55,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:55,586 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,586 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:55,586 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:55,586 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:55,586 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:55,588 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,588 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,588 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,588 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:55,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64505935[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:55,589 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:55,589 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,590 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,590 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:55,590 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:55,590 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:55,590 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,590 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:55,592 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:55,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:55,594 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:55,594 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:55,594 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:55,594 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:55,594 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:55,596 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,596 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,596 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,596 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,597 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,599 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,599 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,599 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:55,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:55,599 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:55,599 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,599 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,599 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:55,599 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:55,599 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:55,599 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,599 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:55,599 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:55,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:55,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:55,609 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839015601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839015608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839015609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839015609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839015658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,699 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:55,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:55,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:55,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:55,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:55,703 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:55,703 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:55,703 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:55,703 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:55,703 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,703 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,704 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,704 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:55,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:55,704 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,704 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:55,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:55,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:55,706 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:55,706 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:55,706 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:55,706 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:55,706 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:55,706 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,706 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:55,706 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:55,706 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,706 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:55,706 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:55,706 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,707 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:55,707 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:55,709 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,709 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,709 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,709 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:55,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:55,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@786e5f0c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:55,711 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:55,711 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,712 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,712 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:55,712 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:55,712 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,712 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:55,712 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:55,713 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,713 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,713 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:55,713 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839015710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:55,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:55,715 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:55,716 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:55,716 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:55,716 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:55,716 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:55,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,718 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,719 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839015713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839015716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,722 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,722 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,722 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:55,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:55,722 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:55,722 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:55,722 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,722 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,722 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,722 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,722 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,722 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:55,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:55,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:55,722 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:55,722 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:55,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:55,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839015716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,822 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:55,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:55,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:55,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:55,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:55,826 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:55,826 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:55,826 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:55,826 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:55,826 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,827 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,828 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:55,828 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:55,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:55,828 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,828 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:55,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:55,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:55,830 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:55,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:55,830 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:55,830 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:55,830 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:55,831 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:55,831 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:55,831 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,831 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:55,831 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:55,831 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,831 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:55,831 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:55,831 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,831 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:55,832 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:55,833 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,833 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,833 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,833 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:55,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39c61527[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:55,835 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:55,835 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,836 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,836 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:55,836 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:55,836 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:55,836 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,836 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:55,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,842 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:55,842 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,842 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:55,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:55,848 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:55,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:55,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:55,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:55,848 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:55,850 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,850 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,850 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:55,850 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,851 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,854 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:55,854 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:55,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:55,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:55,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,855 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:55,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:55,856 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:55,856 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:55,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:55,871 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839015863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839015916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839015921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839015921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:55,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839015924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/c1dd2ff904d74b14a8b0d6c36bd5c285 2023-03-26 13:55:55,953 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/fdf6c9953c814283bad56cc1fbfcef37 2023-03-26 13:55:55,956 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:55,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:55,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:55,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:55,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:55,961 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:55,961 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:55,961 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:55,961 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:55,961 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,961 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,963 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,963 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:55,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:55,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:55,964 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,964 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:55,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:55,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:55,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:55,971 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:55,971 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:55,971 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:55,972 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:55,972 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:55,972 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,972 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:55,972 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:55,972 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,972 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:55,973 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:55,973 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,973 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:55,973 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:55,973 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=438 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/4d070fe488bd4020870217d55e309132 2023-03-26 13:55:55,974 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,974 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,975 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:55,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:55,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15415759[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:55,977 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:55,977 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,977 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/c1dd2ff904d74b14a8b0d6c36bd5c285 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/c1dd2ff904d74b14a8b0d6c36bd5c285 2023-03-26 13:55:55,978 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,979 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:55,979 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:55,979 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:55,980 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:55,980 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:55,980 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:55,981 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,981 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:55,982 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:55,982 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:55,982 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:55,982 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:55,982 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:55,982 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:55,983 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/c1dd2ff904d74b14a8b0d6c36bd5c285, entries=250, sequenceid=438, filesize=16.7 K 2023-03-26 13:55:55,983 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/fdf6c9953c814283bad56cc1fbfcef37 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/fdf6c9953c814283bad56cc1fbfcef37 2023-03-26 13:55:55,985 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,985 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:55,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,985 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:55,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:55,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,986 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,988 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/fdf6c9953c814283bad56cc1fbfcef37, entries=150, sequenceid=438, filesize=11.9 K 2023-03-26 13:55:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:55,989 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,989 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/4d070fe488bd4020870217d55e309132 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/4d070fe488bd4020870217d55e309132 2023-03-26 13:55:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,992 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,992 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,992 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:55,992 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:55,992 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:55,992 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:55,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:55,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:55,992 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:55,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:55,992 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,992 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:55,992 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:55,992 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:55,992 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:55,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:55,993 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/4d070fe488bd4020870217d55e309132, entries=150, sequenceid=438, filesize=11.9 K 2023-03-26 13:55:55,994 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 2990228151772e454b24e80da60e1bb2 in 461ms, sequenceid=438, compaction requested=true 2023-03-26 13:55:55,994 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:55,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:55,994 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:55,994 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:55,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:55,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:55,995 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42358 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:55,995 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/A is initiating minor compaction (all files) 2023-03-26 13:55:55,995 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:55,995 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/A in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,995 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/B is initiating minor compaction (all files) 2023-03-26 13:55:55,995 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/B in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:55,995 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/4cbc3880fba546b3845e6c8756d6fc27, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/f05690db23ef4f7c9c9999096747a45b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/c1dd2ff904d74b14a8b0d6c36bd5c285] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=41.4 K 2023-03-26 13:55:55,995 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a720ae392ac14e519394b64413549435, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/f8860e3621684134a40aaa8ecdb3fea0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/fdf6c9953c814283bad56cc1fbfcef37] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=36.6 K 2023-03-26 13:55:55,996 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 4cbc3880fba546b3845e6c8756d6fc27, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1679838954840 2023-03-26 13:55:55,996 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting a720ae392ac14e519394b64413549435, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1679838954840 2023-03-26 13:55:55,996 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting f05690db23ef4f7c9c9999096747a45b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1679838954900 2023-03-26 13:55:55,996 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting f8860e3621684134a40aaa8ecdb3fea0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1679838954900 2023-03-26 13:55:55,997 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting c1dd2ff904d74b14a8b0d6c36bd5c285, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1679838955527 2023-03-26 13:55:55,997 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting fdf6c9953c814283bad56cc1fbfcef37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1679838955530 2023-03-26 13:55:56,005 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#A#compaction#525 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:56,015 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#B#compaction#526 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:56,029 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/7ebff05f3a68430fbb3d80218e65be09 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/7ebff05f3a68430fbb3d80218e65be09 2023-03-26 13:55:56,033 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/A of 2990228151772e454b24e80da60e1bb2 into 7ebff05f3a68430fbb3d80218e65be09(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:56,033 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:56,033 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/A, priority=13, startTime=1679838955994; duration=0sec 2023-03-26 13:55:56,033 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:56,033 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:55:56,034 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:55:56,034 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/C is initiating minor compaction (all files) 2023-03-26 13:55:56,034 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/C in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:56,034 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/9d8ae4309eb64ee9adc328baa44c7ee7, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/8aade3946dd04b8b8605e7f23dce51d0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/5def744979d94ed5bf446bdb4001d395, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/4d070fe488bd4020870217d55e309132] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=48.4 K 2023-03-26 13:55:56,035 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 9d8ae4309eb64ee9adc328baa44c7ee7, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=376, earliestPutTs=1679838954206 2023-03-26 13:55:56,035 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 8aade3946dd04b8b8605e7f23dce51d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1679838954840 2023-03-26 13:55:56,035 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 5def744979d94ed5bf446bdb4001d395, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1679838954900 2023-03-26 13:55:56,036 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 4d070fe488bd4020870217d55e309132, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1679838955530 2023-03-26 13:55:56,043 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/ab638aef9d7f4e6782985accd53b2e11 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/ab638aef9d7f4e6782985accd53b2e11 2023-03-26 13:55:56,047 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#C#compaction#527 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:56,051 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/B of 2990228151772e454b24e80da60e1bb2 into ab638aef9d7f4e6782985accd53b2e11(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:56,051 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:56,051 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/B, priority=13, startTime=1679838955994; duration=0sec 2023-03-26 13:55:56,051 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:56,056 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/6083ba607b094299b0710904de1a0100 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/6083ba607b094299b0710904de1a0100 2023-03-26 13:55:56,059 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2990228151772e454b24e80da60e1bb2/C of 2990228151772e454b24e80da60e1bb2 into 6083ba607b094299b0710904de1a0100(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:56,060 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:56,060 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/C, priority=12, startTime=1679838955994; duration=0sec 2023-03-26 13:55:56,060 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:56,093 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:56,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:56,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:56,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:56,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:56,095 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:56,095 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:56,095 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:56,095 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:56,096 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,096 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,097 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,097 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:56,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:56,099 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,099 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,099 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:56,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:56,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:56,100 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:56,100 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:56,100 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:56,100 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-03-26 13:55:56,101 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:56,101 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:56,101 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:56,101 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:56,101 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:56,101 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:56,109 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/1733113733cc43339e8e190e45f8edfd 2023-03-26 13:55:56,125 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/39f6865fa7d146018a395c851495fab9 2023-03-26 13:55:56,138 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/2d38473ba1b14282b095930c63ef945a 2023-03-26 13:55:56,141 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/1733113733cc43339e8e190e45f8edfd as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/1733113733cc43339e8e190e45f8edfd 2023-03-26 13:55:56,145 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/1733113733cc43339e8e190e45f8edfd, entries=150, sequenceid=475, filesize=11.9 K 2023-03-26 13:55:56,146 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/39f6865fa7d146018a395c851495fab9 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/39f6865fa7d146018a395c851495fab9 2023-03-26 13:55:56,148 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/39f6865fa7d146018a395c851495fab9, entries=150, sequenceid=475, filesize=11.9 K 2023-03-26 13:55:56,149 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/2d38473ba1b14282b095930c63ef945a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/2d38473ba1b14282b095930c63ef945a 2023-03-26 13:55:56,152 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/2d38473ba1b14282b095930c63ef945a, entries=150, sequenceid=475, filesize=11.9 K 2023-03-26 13:55:56,153 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=0 B/0 for 2990228151772e454b24e80da60e1bb2 in 53ms, sequenceid=475, compaction requested=false 2023-03-26 13:55:56,153 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:56,153 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:56,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:56,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:56,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:56,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:56,155 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,155 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,155 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,155 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:56,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:56,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@733e2d35[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:56,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:56,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,159 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,159 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:56,159 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:56,159 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:56,159 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,159 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:56,160 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:56,160 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:56,160 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:56,161 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:56,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:56,163 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:56,164 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:56,164 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:56,164 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:56,164 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:56,165 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,165 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:56,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,165 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:56,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,166 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,167 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,169 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:56,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:56,169 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:56,169 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:56,169 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,169 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,169 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,169 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,170 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,170 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:56,169 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:56,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:56,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:56,169 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:56,170 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:56,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:56,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:56,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:56,185 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:55:56,185 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:56,185 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:56,185 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:56,185 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:56,185 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:56,185 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:56,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838956232, max timestamp=1679838956240Num uniques 859; 2023-03-26 13:55:56,241 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838956232, max timestamp=1679838956240Num uniques 859; 2023-03-26 13:55:56,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679838956232, max timestamp=1679838956240Num uniques 859; 2023-03-26 13:55:56,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:55:56,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:55:56,242 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 13:55:56,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839016255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839016255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839016259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839016260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839016261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,270 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:56,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:56,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:56,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:56,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:56,273 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:56,273 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:56,273 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:56,273 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:56,273 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,273 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,275 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:56,275 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:56,276 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,276 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:56,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:56,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:56,277 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:56,277 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:56,277 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:56,277 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:56,277 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,278 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:56,278 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:56,279 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,279 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,279 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:56,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79228a61[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:56,281 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:56,281 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,282 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:56,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:56,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:56,282 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,283 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:56,284 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:56,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:56,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:56,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:56,286 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:56,286 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:56,286 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:56,286 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:56,286 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:56,289 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,289 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,289 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:56,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:56,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,290 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,292 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:56,297 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:56,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:56,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:56,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:56,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:56,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:56,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,297 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:56,298 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:56,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:56,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:56,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839016362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839016362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839016362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,368 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839016366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839016367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,398 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:56,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:56,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:56,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:56,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:56,401 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:56,401 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:56,401 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:56,401 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:56,401 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,401 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,402 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,402 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:56,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:56,403 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,403 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:56,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:56,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:56,405 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:56,405 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:56,405 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:56,405 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:56,405 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:56,405 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:56,405 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:56,405 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:56,405 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:56,405 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:56,405 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:56,405 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,405 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:56,405 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:56,407 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,407 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,407 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,407 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:56,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7044d74f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:56,408 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:56,408 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,410 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,411 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:56,411 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:56,411 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:56,412 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:56,412 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,412 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:56,412 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:56,412 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:56,412 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:56,412 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:56,412 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:56,412 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:56,412 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:56,412 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:56,414 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,414 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,414 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:56,414 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,415 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,417 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,417 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,417 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,417 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:56,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:56,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:56,417 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,417 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:56,418 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,418 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:56,418 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:56,418 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:56,418 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:56,418 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:56,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:56,518 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:56,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:56,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:56,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:56,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:56,522 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:56,522 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:56,522 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:56,522 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:56,522 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,522 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,524 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,524 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,525 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:56,532 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:56,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:56,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:56,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:56,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:56,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:56,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:56,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:56,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:56,540 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:56,540 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:56,540 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:56,540 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:56,540 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:56,540 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:56,540 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:56,540 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:56,540 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,540 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:56,540 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:56,541 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,542 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,542 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,542 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:56,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f20ad81[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:56,543 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:56,543 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,544 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,544 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:56,544 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:56,544 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:56,544 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,544 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:56,546 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,546 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:56,546 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:56,546 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:56,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:56,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:56,549 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:56,549 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:56,549 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:56,549 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:56,549 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:56,551 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,551 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,551 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:56,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,552 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,554 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,556 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,556 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,556 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:56,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:56,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:56,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:56,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:56,556 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:56,556 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:56,556 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,556 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:56,556 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:56,556 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:56,556 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,557 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,557 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839016569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839016569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,573 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839016569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839016570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,574 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839016572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,592 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/aac3822f7dc64578848b4c7163c9d3c8 2023-03-26 13:55:56,657 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:56,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:56,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:56,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:56,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:56,662 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:56,662 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:56,662 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:56,662 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:56,662 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,662 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,663 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:56,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:56,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:56,663 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,664 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,664 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:56,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,664 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:56,664 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:56,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:56,665 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:56,665 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:56,665 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:56,665 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:56,665 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:56,665 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:56,665 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:56,665 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:56,665 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:56,665 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:56,665 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:56,665 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,665 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:56,665 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:56,667 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:56,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6df572d5[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:56,669 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:56,669 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:56,671 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,671 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,672 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:56,672 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:56,672 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:56,672 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:56,673 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,673 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:56,673 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,673 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:56,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:56,676 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:56,677 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:56,677 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:56,677 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:56,677 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:56,678 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,678 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,678 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,679 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,680 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,686 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:56,687 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:56,687 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:56,687 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,687 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:56,687 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:56,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:56,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:56,687 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,687 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,687 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,687 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:56,687 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:56,687 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:56,787 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:56,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:56,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:56,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:56,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:56,791 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:56,791 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:56,791 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:56,791 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:56,791 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,791 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,792 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:56,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:56,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:56,793 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,793 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,793 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:56,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:56,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:56,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:56,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:56,794 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:56,794 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:56,794 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:56,795 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:56,795 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:56,795 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:56,795 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:56,795 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:56,795 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:56,795 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:56,795 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:56,795 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,795 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:56,795 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:56,797 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,797 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,797 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,797 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:56,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:56,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@699c88[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:56,799 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:56,799 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,801 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,801 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:56,801 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:56,801 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:56,801 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,801 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:56,803 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:56,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:56,803 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:56,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:56,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:56,807 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:56,807 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:56,807 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:56,807 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:56,807 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:56,809 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,809 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:56,809 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,810 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,812 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:56,814 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:56,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:56,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:56,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:56,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:56,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:56,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:56,815 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:56,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:56,815 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:56,815 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:56,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839016875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839016875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839016875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:56,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839016876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839016876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,915 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:56,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:56,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:56,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:56,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:56,919 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:56,919 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:56,919 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:56,919 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:56,919 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,919 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,921 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:56,921 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:56,922 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,922 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:56,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:56,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:56,923 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:56,923 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:56,923 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:56,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:56,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:56,924 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:56,924 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:56,924 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:56,924 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:56,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:56,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:56,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:56,925 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:56,926 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,927 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,927 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:56,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5641cd65[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:56,928 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:56,928 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,930 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:56,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:56,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:56,931 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,931 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:56,932 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:56,932 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:56,932 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:56,932 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:56,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:56,934 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:56,934 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:56,934 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:56,934 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:56,935 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:56,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:56,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:56,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:56,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:56,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,939 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:56,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:56,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:56,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,940 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:56,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:56,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:56,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:56,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:56,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:56,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:56,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:56,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:56,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:56,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:56,944 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:56,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:56,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:56,945 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:56,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:56,945 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:57,003 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/52fdf3c4a4b44b1bbe4c26903bb9f7b4 2023-03-26 13:55:57,045 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:57,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:57,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:57,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:57,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:57,050 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:57,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:57,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:57,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:57,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,052 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:57,052 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:57,052 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,052 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:57,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:57,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:57,053 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:57,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:57,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:57,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:57,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:57,054 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:57,056 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:57,056 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,056 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:57,056 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:57,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:57,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c673e89[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:57,058 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:57,058 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,060 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,061 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:57,061 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:57,061 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:57,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:57,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:57,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:57,062 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,063 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:57,063 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:57,063 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:57,063 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:57,063 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:57,063 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:57,063 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:57,065 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:57,065 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:57,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,065 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:57,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:57,065 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:57,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:57,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:57,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:57,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:57,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:57,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:57,074 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:57,075 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,075 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,075 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:57,075 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:57,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:57,175 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:57,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:57,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:57,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:57,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:57,179 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:57,179 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:57,179 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:57,179 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:57,180 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,180 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,184 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:57,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:57,184 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,185 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,185 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:57,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:57,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:57,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:57,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:57,187 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:57,188 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:57,188 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:57,188 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:57,188 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:57,188 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:57,188 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:57,188 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:57,188 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:57,188 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,188 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:57,188 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:57,192 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,193 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:57,193 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,193 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:57,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:57,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a42900a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:57,194 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:57,195 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,196 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:57,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:57,196 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,196 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:57,197 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:57,198 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:57,198 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:57,198 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:57,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:57,201 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:57,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:57,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:57,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:57,202 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:57,205 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,205 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:57,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:57,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,206 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:57,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:57,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:57,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,209 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:57,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,211 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:57,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,212 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,213 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:57,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:57,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:57,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:57,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:57,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:57,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:57,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,213 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,213 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:57,213 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:57,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:57,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:57,314 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:57,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:57,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:57,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:57,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:57,317 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:57,317 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:57,317 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:57,317 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:57,320 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,320 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,321 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,321 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:57,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:57,322 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,322 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:57,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:57,323 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:57,323 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:57,323 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:57,323 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:57,323 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:57,323 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:57,323 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:57,323 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:57,324 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:57,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:57,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:57,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:57,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:57,325 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:57,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:57,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:57,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3683de16[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:57,327 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:57,327 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,329 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,329 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:57,329 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:57,329 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:57,329 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,329 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:57,331 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,331 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:57,331 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:57,331 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:57,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:57,333 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:57,333 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:57,333 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:57,333 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:57,333 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:57,335 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,335 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,335 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:57,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,335 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,338 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,339 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,339 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,339 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:57,339 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:57,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:57,339 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:57,339 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:57,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:57,339 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,340 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:57,340 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,340 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:57,340 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:57,340 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,388 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:57,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839017383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:57,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839017386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:57,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839017387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:57,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839017387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,390 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:57,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839017387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,433 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/ee03866624884317a0a1b77a04a03adc 2023-03-26 13:55:57,436 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/aac3822f7dc64578848b4c7163c9d3c8 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/aac3822f7dc64578848b4c7163c9d3c8 2023-03-26 13:55:57,439 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/aac3822f7dc64578848b4c7163c9d3c8, entries=250, sequenceid=485, filesize=16.7 K 2023-03-26 13:55:57,441 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:57,441 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/52fdf3c4a4b44b1bbe4c26903bb9f7b4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/52fdf3c4a4b44b1bbe4c26903bb9f7b4 2023-03-26 13:55:57,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:57,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:57,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:57,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:57,444 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:57,444 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:57,444 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:57,444 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:57,444 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,444 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,445 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/52fdf3c4a4b44b1bbe4c26903bb9f7b4, entries=150, sequenceid=485, filesize=11.9 K 2023-03-26 13:55:57,445 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/ee03866624884317a0a1b77a04a03adc as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ee03866624884317a0a1b77a04a03adc 2023-03-26 13:55:57,447 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,447 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:57,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:57,447 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,447 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:57,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:57,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:57,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:57,448 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:57,448 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:57,448 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:57,449 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:57,449 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:57,449 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:57,449 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:57,449 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:57,449 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:57,449 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:57,449 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:57,449 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ee03866624884317a0a1b77a04a03adc, entries=150, sequenceid=485, filesize=11.9 K 2023-03-26 13:55:57,449 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,449 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:57,449 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:57,450 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2990228151772e454b24e80da60e1bb2 in 1265ms, sequenceid=485, compaction requested=true 2023-03-26 13:55:57,450 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:57,450 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:57,450 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:57,450 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:57,450 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:57,450 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:57,451 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:57,451 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,451 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:57,451 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/B is initiating minor compaction (all files) 2023-03-26 13:55:57,451 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/B in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:57,451 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42460 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:57,452 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/ab638aef9d7f4e6782985accd53b2e11, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/39f6865fa7d146018a395c851495fab9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/52fdf3c4a4b44b1bbe4c26903bb9f7b4] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=36.7 K 2023-03-26 13:55:57,452 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/A is initiating minor compaction (all files) 2023-03-26 13:55:57,451 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,452 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/A in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:57,452 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/7ebff05f3a68430fbb3d80218e65be09, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/1733113733cc43339e8e190e45f8edfd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/aac3822f7dc64578848b4c7163c9d3c8] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=41.5 K 2023-03-26 13:55:57,452 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:57,452 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting ab638aef9d7f4e6782985accd53b2e11, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1679838955530 2023-03-26 13:55:57,452 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 7ebff05f3a68430fbb3d80218e65be09, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1679838955530 2023-03-26 13:55:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:57,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:57,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e710746[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:57,453 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:57,453 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,454 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 39f6865fa7d146018a395c851495fab9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1679838955563 2023-03-26 13:55:57,454 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 1733113733cc43339e8e190e45f8edfd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1679838955563 2023-03-26 13:55:57,454 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 52fdf3c4a4b44b1bbe4c26903bb9f7b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1679838956180 2023-03-26 13:55:57,454 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,454 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting aac3822f7dc64578848b4c7163c9d3c8, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1679838956173 2023-03-26 13:55:57,454 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,454 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:57,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,455 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:57,455 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:57,455 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:57,456 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,456 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:57,456 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:57,456 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,459 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#B#compaction#534 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:57,460 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#A#compaction#535 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:57,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:57,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:57,461 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:57,461 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:57,461 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:57,461 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:57,462 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:57,464 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,464 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:57,464 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:57,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:57,464 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,472 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,477 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,477 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,477 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,477 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,477 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,477 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:57,477 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:57,477 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:57,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:57,477 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:57,478 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:57,478 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:57,478 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:57,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:57,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:57,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:57,480 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/b26009bb60a84890b8daec590d213b5b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b26009bb60a84890b8daec590d213b5b 2023-03-26 13:55:57,483 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/a077e121521b42cf99b199e7716a4f54 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a077e121521b42cf99b199e7716a4f54 2023-03-26 13:55:57,485 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/A of 2990228151772e454b24e80da60e1bb2 into b26009bb60a84890b8daec590d213b5b(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:57,485 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:57,485 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/A, priority=13, startTime=1679838957450; duration=0sec 2023-03-26 13:55:57,485 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:57,485 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:57,486 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:57,486 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/C is initiating minor compaction (all files) 2023-03-26 13:55:57,486 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/C in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:57,486 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/6083ba607b094299b0710904de1a0100, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/2d38473ba1b14282b095930c63ef945a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ee03866624884317a0a1b77a04a03adc] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=36.6 K 2023-03-26 13:55:57,487 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 6083ba607b094299b0710904de1a0100, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=438, earliestPutTs=1679838955530 2023-03-26 13:55:57,487 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 2d38473ba1b14282b095930c63ef945a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1679838955563 2023-03-26 13:55:57,487 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/B of 2990228151772e454b24e80da60e1bb2 into a077e121521b42cf99b199e7716a4f54(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:57,487 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:57,487 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting ee03866624884317a0a1b77a04a03adc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1679838956180 2023-03-26 13:55:57,487 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/B, priority=13, startTime=1679838957450; duration=0sec 2023-03-26 13:55:57,488 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:57,493 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#C#compaction#536 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:57,501 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/ea8ef92dc9e74f51acd8e5f896f8e8e1 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ea8ef92dc9e74f51acd8e5f896f8e8e1 2023-03-26 13:55:57,505 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/C of 2990228151772e454b24e80da60e1bb2 into ea8ef92dc9e74f51acd8e5f896f8e8e1(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:57,505 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:57,505 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/C, priority=13, startTime=1679838957450; duration=0sec 2023-03-26 13:55:57,505 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:57,578 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:57,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:57,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:57,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:57,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:57,581 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:57,581 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:57,581 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:57,581 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:57,581 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,581 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,582 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,582 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:57,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:57,583 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:57,583 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:57,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:57,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:57,584 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:57,584 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:57,584 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:57,584 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:57,584 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:57,584 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:57,584 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:57,584 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:57,584 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-26 13:55:57,584 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:57,584 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:57,585 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:57,585 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:57,585 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:57,585 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:55:57,604 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/ed508833cd7547099b648686af8e45b7 2023-03-26 13:55:58,019 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/0710c44047544d44befa93f9e0197309 2023-03-26 13:55:58,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:58,395 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:58,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839018408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839018409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839018411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839018411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839018411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,437 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=523 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/4eb053d4a2bd4b5daf4d8041d3671783 2023-03-26 13:55:58,441 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/ed508833cd7547099b648686af8e45b7 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/ed508833cd7547099b648686af8e45b7 2023-03-26 13:55:58,446 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/ed508833cd7547099b648686af8e45b7, entries=150, sequenceid=523, filesize=11.9 K 2023-03-26 13:55:58,448 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/0710c44047544d44befa93f9e0197309 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/0710c44047544d44befa93f9e0197309 2023-03-26 13:55:58,451 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/0710c44047544d44befa93f9e0197309, entries=150, sequenceid=523, filesize=11.9 K 2023-03-26 13:55:58,452 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/4eb053d4a2bd4b5daf4d8041d3671783 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/4eb053d4a2bd4b5daf4d8041d3671783 2023-03-26 13:55:58,464 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/4eb053d4a2bd4b5daf4d8041d3671783, entries=150, sequenceid=523, filesize=11.9 K 2023-03-26 13:55:58,465 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=60.38 KB/61830 for 2990228151772e454b24e80da60e1bb2 in 881ms, sequenceid=523, compaction requested=false 2023-03-26 13:55:58,465 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:58,465 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:58,465 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:58,465 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:58,465 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,465 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:58,465 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:58,468 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,468 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:58,468 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,468 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:58,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:58,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:58,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b67ef77[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:58,470 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:58,470 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,472 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,472 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:58,472 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:58,472 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:58,472 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,472 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:58,474 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,474 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:58,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:58,474 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:58,474 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:58,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:58,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:58,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:58,477 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:58,477 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:58,477 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:58,477 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:58,477 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:58,479 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,479 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,479 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:58,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,480 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:58,480 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:58,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:58,489 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,489 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:58,489 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:58,489 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:58,489 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:58,489 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:58,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:58,489 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,489 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,489 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:58,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:58,489 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:58,489 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:58,489 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:58,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:58,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:58,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:58,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:58,516 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-26 13:55:58,517 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:58,517 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:58,517 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:58,517 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:58,517 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:58,517 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:58,524 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/f9b1f18fa7324236ba6f266cd409df5e 2023-03-26 13:55:58,538 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/08f9e2dae5084281a2c739bfaacfdfbe 2023-03-26 13:55:58,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839018536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,549 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=536 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/fecf2a19a3ec41b5940bd333e8706325 2023-03-26 13:55:58,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839018544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839018544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839018545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839018545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,554 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/f9b1f18fa7324236ba6f266cd409df5e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/f9b1f18fa7324236ba6f266cd409df5e 2023-03-26 13:55:58,557 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/f9b1f18fa7324236ba6f266cd409df5e, entries=200, sequenceid=536, filesize=14.3 K 2023-03-26 13:55:58,558 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/08f9e2dae5084281a2c739bfaacfdfbe as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/08f9e2dae5084281a2c739bfaacfdfbe 2023-03-26 13:55:58,561 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/08f9e2dae5084281a2c739bfaacfdfbe, entries=150, sequenceid=536, filesize=11.9 K 2023-03-26 13:55:58,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/fecf2a19a3ec41b5940bd333e8706325 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/fecf2a19a3ec41b5940bd333e8706325 2023-03-26 13:55:58,566 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/fecf2a19a3ec41b5940bd333e8706325, entries=150, sequenceid=536, filesize=11.9 K 2023-03-26 13:55:58,567 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 2990228151772e454b24e80da60e1bb2 in 51ms, sequenceid=536, compaction requested=true 2023-03-26 13:55:58,567 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:58,567 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:58,567 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:58,567 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:58,567 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:58,567 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:58,568 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:58,568 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:58,568 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/A is initiating minor compaction (all files) 2023-03-26 13:55:58,568 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/B is initiating minor compaction (all files) 2023-03-26 13:55:58,568 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/A in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:58,568 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/B in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:58,568 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b26009bb60a84890b8daec590d213b5b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/ed508833cd7547099b648686af8e45b7, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/f9b1f18fa7324236ba6f266cd409df5e] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=39.2 K 2023-03-26 13:55:58,568 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a077e121521b42cf99b199e7716a4f54, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/0710c44047544d44befa93f9e0197309, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/08f9e2dae5084281a2c739bfaacfdfbe] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=36.8 K 2023-03-26 13:55:58,569 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting b26009bb60a84890b8daec590d213b5b, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1679838956180 2023-03-26 13:55:58,569 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting a077e121521b42cf99b199e7716a4f54, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1679838956180 2023-03-26 13:55:58,569 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting ed508833cd7547099b648686af8e45b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1679838956254 2023-03-26 13:55:58,569 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 0710c44047544d44befa93f9e0197309, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1679838956254 2023-03-26 13:55:58,569 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting f9b1f18fa7324236ba6f266cd409df5e, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1679838958408 2023-03-26 13:55:58,569 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 08f9e2dae5084281a2c739bfaacfdfbe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1679838958408 2023-03-26 13:55:58,577 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#B#compaction#544 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:58,577 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#A#compaction#543 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:58,589 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:58,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:58,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:58,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:58,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:58,599 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:58,599 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:58,599 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:58,599 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:58,600 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,600 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:58,602 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:58,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:58,602 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:58,602 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,602 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:58,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:58,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:58,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:58,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:58,603 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:58,603 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:58,603 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:58,604 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/2be6be788f764b7ea91870a4c929299d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/2be6be788f764b7ea91870a4c929299d 2023-03-26 13:55:58,604 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:58,604 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:58,604 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:58,604 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:58,604 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-26 13:55:58,604 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:58,604 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:58,605 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:58,605 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:58,605 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:58,605 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:58,606 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/088fa57a1d054400a8711ac2e55f4c14 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/088fa57a1d054400a8711ac2e55f4c14 2023-03-26 13:55:58,612 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/A of 2990228151772e454b24e80da60e1bb2 into 2be6be788f764b7ea91870a4c929299d(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:58,612 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:58,612 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/A, priority=13, startTime=1679838958567; duration=0sec 2023-03-26 13:55:58,612 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:58,612 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:58,613 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/B of 2990228151772e454b24e80da60e1bb2 into 088fa57a1d054400a8711ac2e55f4c14(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:58,613 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:58,613 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/B, priority=13, startTime=1679838958567; duration=0sec 2023-03-26 13:55:58,613 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:58,614 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:58,614 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/C is initiating minor compaction (all files) 2023-03-26 13:55:58,614 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/C in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:58,614 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ea8ef92dc9e74f51acd8e5f896f8e8e1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/4eb053d4a2bd4b5daf4d8041d3671783, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/fecf2a19a3ec41b5940bd333e8706325] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=36.7 K 2023-03-26 13:55:58,615 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting ea8ef92dc9e74f51acd8e5f896f8e8e1, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1679838956180 2023-03-26 13:55:58,615 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 4eb053d4a2bd4b5daf4d8041d3671783, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=523, earliestPutTs=1679838956254 2023-03-26 13:55:58,615 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting fecf2a19a3ec41b5940bd333e8706325, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1679838958408 2023-03-26 13:55:58,618 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/13ca310abcd246ffa8deffab998dcb5a 2023-03-26 13:55:58,620 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#C#compaction#546 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:58,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:58,650 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:58,657 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/94efc9a58fe94742844b8ff16935b07c 2023-03-26 13:55:58,659 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/aff090b84a084216811f040d8e0462e3 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/aff090b84a084216811f040d8e0462e3 2023-03-26 13:55:58,667 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/C of 2990228151772e454b24e80da60e1bb2 into aff090b84a084216811f040d8e0462e3(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:58,667 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:58,667 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/C, priority=13, startTime=1679838958567; duration=0sec 2023-03-26 13:55:58,668 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:58,669 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839018663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,674 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839018668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,675 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839018669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839018670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839018671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,690 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/b51754b36cf14eb7b88cbb6823e6699c 2023-03-26 13:55:58,693 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/13ca310abcd246ffa8deffab998dcb5a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/13ca310abcd246ffa8deffab998dcb5a 2023-03-26 13:55:58,704 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/13ca310abcd246ffa8deffab998dcb5a, entries=150, sequenceid=555, filesize=11.9 K 2023-03-26 13:55:58,705 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/94efc9a58fe94742844b8ff16935b07c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/94efc9a58fe94742844b8ff16935b07c 2023-03-26 13:55:58,708 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/94efc9a58fe94742844b8ff16935b07c, entries=150, sequenceid=555, filesize=11.9 K 2023-03-26 13:55:58,709 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/b51754b36cf14eb7b88cbb6823e6699c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b51754b36cf14eb7b88cbb6823e6699c 2023-03-26 13:55:58,712 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b51754b36cf14eb7b88cbb6823e6699c, entries=150, sequenceid=555, filesize=11.9 K 2023-03-26 13:55:58,713 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 2990228151772e454b24e80da60e1bb2 in 109ms, sequenceid=555, compaction requested=false 2023-03-26 13:55:58,713 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:58,713 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:58,713 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:58,713 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:58,713 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,713 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:58,713 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:58,715 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:58,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:58,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:58,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54bf0d1[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:58,716 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:58,716 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,718 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:58,718 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:58,718 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:58,718 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,718 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:58,720 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,720 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:58,720 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:58,720 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:58,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:58,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:58,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:58,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:58,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:58,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:58,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:58,722 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:58,722 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:58,723 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:58,723 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:58,723 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:58,724 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,724 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,725 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:58,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:58,725 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:58,726 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,728 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,728 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:58,728 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:58,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:58,728 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:58,728 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,729 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,729 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,729 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:58,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:58,728 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:58,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:58,728 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:58,729 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:58,729 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:58,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:58,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:58,772 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-26 13:55:58,772 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:58,773 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:58,773 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:58,773 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:58,773 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:58,773 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:58,780 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/caba37a5685548db9eedf4ed87bbc18d 2023-03-26 13:55:58,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839018794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839018794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839018795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839018796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839018797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,804 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/9484019c4e4043e3b535ed74deca8e99 2023-03-26 13:55:58,829 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:58,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:58,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:58,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:58,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:58,834 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:58,834 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:58,834 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:58,834 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:58,835 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,835 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:58,836 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:58,836 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:58,836 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,836 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:58,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:58,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:58,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:58,838 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:58,838 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:58,838 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:58,838 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:58,838 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:58,838 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:58,838 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:58,838 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:58,838 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:58,838 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:58,838 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:58,838 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:58,838 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:58,838 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,838 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:58,839 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:58,844 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,844 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:58,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:58,845 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,845 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:58,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:58,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:58,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:58,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:58,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:58,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d90ed00[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:58,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:58,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,849 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,849 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:58,849 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,849 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:58,849 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:58,850 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:58,851 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:58,851 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,851 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:58,851 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:58,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:58,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:58,853 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:58,853 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:58,853 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:58,854 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:58,854 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:58,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:58,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:58,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:58,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:58,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:58,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,857 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:58,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:58,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:58,859 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:58,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:58,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:58,863 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:58,863 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:58,863 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,863 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:58,863 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:58,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:58,863 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:58,864 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,864 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:58,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:58,864 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,864 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,864 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:58,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:58,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:55:58,864 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:58,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839018899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839018899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839018903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839018903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:58,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839018904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,965 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:58,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:58,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:58,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:58,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:58,968 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:58,968 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:58,968 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:58,968 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:58,969 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,969 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:58,970 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:58,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:58,970 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:58,971 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,971 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:58,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:58,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:58,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:58,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:58,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:58,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:58,980 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:58,980 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:58,980 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:58,988 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:58,988 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:58,988 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:58,988 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:58,988 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:58,988 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:58,988 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:58,988 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:58,988 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,988 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:58,988 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:58,990 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,990 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:58,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:58,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:58,991 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,991 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:58,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:58,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:58,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b3d84c3[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:58,993 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:58,993 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,994 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:58,994 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:58,994 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:58,994 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:58,995 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:58,995 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:59,000 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:59,000 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:59,000 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:59,000 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:59,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:59,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:59,003 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:59,003 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:59,003 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:59,003 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:59,003 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:59,005 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:59,005 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:59,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:59,005 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:59,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:59,007 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:59,009 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,011 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:59,011 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:59,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:59,011 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,011 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:59,011 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:59,011 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,011 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:59,012 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:59,012 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:59,012 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:59,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:59,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:59,012 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:59,012 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:59,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:59,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:59,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839019102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839019102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,112 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:59,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:59,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:59,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839019112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839019112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839019111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:59,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:59,125 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:59,125 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:59,125 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:59,126 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:59,126 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:59,126 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:59,128 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:59,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:59,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:59,128 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:59,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:59,131 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:59,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:59,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:59,131 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:59,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:59,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:59,144 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:59,144 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:59,144 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:59,148 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:59,148 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:59,148 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:59,148 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:55:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:55:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:55:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:55:59,149 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,149 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:55:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ff5c684[Count = 0] remaining members to acquire global barrier 2023-03-26 13:55:59,151 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:55:59,151 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:59,152 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:59,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:55:59,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:55:59,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:55:59,153 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,153 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:55:59,156 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:59,156 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:55:59,156 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:59,156 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:59,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:59,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:59,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:59,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:55:59,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:55:59,161 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:55:59,162 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:55:59,162 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:55:59,162 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:55:59,162 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:55:59,165 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:55:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:55:59,165 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:59,165 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:59,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:59,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:55:59,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:59,166 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:55:59,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:59,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:59,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,167 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:55:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:55:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,172 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,172 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:59,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:55:59,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:55:59,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:59,172 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:55:59,172 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:55:59,173 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:55:59,172 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:59,173 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:59,173 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,173 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:59,172 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:55:59,173 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:55:59,173 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:59,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:55:59,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:55:59,235 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/4379db71c232469f88c20f8e2ad22a4b 2023-03-26 13:55:59,238 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/caba37a5685548db9eedf4ed87bbc18d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/caba37a5685548db9eedf4ed87bbc18d 2023-03-26 13:55:59,241 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/caba37a5685548db9eedf4ed87bbc18d, entries=200, sequenceid=573, filesize=14.3 K 2023-03-26 13:55:59,242 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/9484019c4e4043e3b535ed74deca8e99 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9484019c4e4043e3b535ed74deca8e99 2023-03-26 13:55:59,246 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9484019c4e4043e3b535ed74deca8e99, entries=150, sequenceid=573, filesize=11.9 K 2023-03-26 13:55:59,246 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/4379db71c232469f88c20f8e2ad22a4b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/4379db71c232469f88c20f8e2ad22a4b 2023-03-26 13:55:59,250 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/4379db71c232469f88c20f8e2ad22a4b, entries=150, sequenceid=573, filesize=11.9 K 2023-03-26 13:55:59,251 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2990228151772e454b24e80da60e1bb2 in 479ms, sequenceid=573, compaction requested=true 2023-03-26 13:55:59,251 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:59,251 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:59,251 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:59,251 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:59,251 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:59,251 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:59,252 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40224 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:59,252 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:59,252 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/A is initiating minor compaction (all files) 2023-03-26 13:55:59,252 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/B is initiating minor compaction (all files) 2023-03-26 13:55:59,252 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/A in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:59,252 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/B in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:59,252 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/2be6be788f764b7ea91870a4c929299d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/13ca310abcd246ffa8deffab998dcb5a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/caba37a5685548db9eedf4ed87bbc18d] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=39.3 K 2023-03-26 13:55:59,252 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/088fa57a1d054400a8711ac2e55f4c14, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/94efc9a58fe94742844b8ff16935b07c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9484019c4e4043e3b535ed74deca8e99] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=36.9 K 2023-03-26 13:55:59,253 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 2be6be788f764b7ea91870a4c929299d, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1679838958408 2023-03-26 13:55:59,253 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 088fa57a1d054400a8711ac2e55f4c14, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1679838958408 2023-03-26 13:55:59,253 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 13ca310abcd246ffa8deffab998dcb5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1679838958535 2023-03-26 13:55:59,253 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 94efc9a58fe94742844b8ff16935b07c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1679838958535 2023-03-26 13:55:59,253 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting caba37a5685548db9eedf4ed87bbc18d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1679838958666 2023-03-26 13:55:59,253 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 9484019c4e4043e3b535ed74deca8e99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1679838958666 2023-03-26 13:55:59,268 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#B#compaction#552 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:59,270 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#A#compaction#553 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:59,273 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:55:59,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:55:59,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:55:59,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:55:59,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:55:59,276 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:55:59,276 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:59,276 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:55:59,276 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:55:59,276 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:59,276 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:59,277 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,277 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:55:59,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:55:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:55:59,278 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,278 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:55:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:59,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:55:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:55:59,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:55:59,280 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:55:59,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:55:59,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:55:59,280 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:55:59,280 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:55:59,280 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:55:59,283 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/3b5484503f694d878381e3b971b8c012 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/3b5484503f694d878381e3b971b8c012 2023-03-26 13:55:59,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:55:59,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:55:59,284 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:59,285 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:55:59,285 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-26 13:55:59,286 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:55:59,286 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:59,286 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:55:59,286 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:59,286 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:55:59,286 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:55:59,287 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/c0da6d26846b4856888091961d4142e9 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/c0da6d26846b4856888091961d4142e9 2023-03-26 13:55:59,290 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/B of 2990228151772e454b24e80da60e1bb2 into 3b5484503f694d878381e3b971b8c012(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:59,290 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:59,290 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/B, priority=13, startTime=1679838959251; duration=0sec 2023-03-26 13:55:59,290 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:55:59,291 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:55:59,291 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:55:59,291 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/C is initiating minor compaction (all files) 2023-03-26 13:55:59,292 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/C in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:55:59,292 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/aff090b84a084216811f040d8e0462e3, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b51754b36cf14eb7b88cbb6823e6699c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/4379db71c232469f88c20f8e2ad22a4b] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=36.8 K 2023-03-26 13:55:59,292 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting aff090b84a084216811f040d8e0462e3, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=536, earliestPutTs=1679838958408 2023-03-26 13:55:59,293 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting b51754b36cf14eb7b88cbb6823e6699c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=555, earliestPutTs=1679838958535 2023-03-26 13:55:59,293 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 4379db71c232469f88c20f8e2ad22a4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1679838958666 2023-03-26 13:55:59,331 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/A of 2990228151772e454b24e80da60e1bb2 into c0da6d26846b4856888091961d4142e9(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:59,331 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:59,331 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/A, priority=13, startTime=1679838959251; duration=0sec 2023-03-26 13:55:59,331 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:59,333 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#C#compaction#555 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:55:59,380 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/57d5682a41014e5eb7601797ba350cfb as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/57d5682a41014e5eb7601797ba350cfb 2023-03-26 13:55:59,387 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2990228151772e454b24e80da60e1bb2/C of 2990228151772e454b24e80da60e1bb2 into 57d5682a41014e5eb7601797ba350cfb(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:55:59,387 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:55:59,387 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/C, priority=13, startTime=1679838959251; duration=0sec 2023-03-26 13:55:59,387 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:55:59,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:55:59,416 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:55:59,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839019427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839019429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839019430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839019434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839019435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839019535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839019536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839019536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839019549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839019549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839019739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839019739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839019739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,751 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/f4bd265f3e4345c78131647fcc563fbd 2023-03-26 13:55:59,754 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839019754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:55:59,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:55:59,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839019754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:00,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839020041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:00,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839020041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:00,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839020042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,057 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:00,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839020056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,061 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:00,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839020061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,164 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/a1ca560748ad4b86b0424c22ab2652ad 2023-03-26 13:56:00,173 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=590 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/5249d344b90b46fa889e5241e4957de2 2023-03-26 13:56:00,179 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/f4bd265f3e4345c78131647fcc563fbd as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/f4bd265f3e4345c78131647fcc563fbd 2023-03-26 13:56:00,184 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/f4bd265f3e4345c78131647fcc563fbd, entries=150, sequenceid=590, filesize=11.9 K 2023-03-26 13:56:00,185 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/a1ca560748ad4b86b0424c22ab2652ad as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a1ca560748ad4b86b0424c22ab2652ad 2023-03-26 13:56:00,188 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a1ca560748ad4b86b0424c22ab2652ad, entries=150, sequenceid=590, filesize=11.9 K 2023-03-26 13:56:00,188 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/5249d344b90b46fa889e5241e4957de2 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/5249d344b90b46fa889e5241e4957de2 2023-03-26 13:56:00,191 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/5249d344b90b46fa889e5241e4957de2, entries=150, sequenceid=590, filesize=11.9 K 2023-03-26 13:56:00,192 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2990228151772e454b24e80da60e1bb2 in 907ms, sequenceid=590, compaction requested=false 2023-03-26 13:56:00,192 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:56:00,192 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:00,192 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:00,192 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:00,192 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,192 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:00,192 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:00,194 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:00,194 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:00,194 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:00,194 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:00,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:00,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@456260d3[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:00,196 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:00,196 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:00,198 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:00,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:00,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:00,198 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:00,199 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:00,199 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:00,199 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,199 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:00,200 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,200 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:00,200 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:00,200 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:00,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:00,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:00,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:00,202 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:00,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:00,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:00,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:00,202 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:00,204 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:00,204 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:00,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:00,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:00,204 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:00,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:00,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:00,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:00,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:00,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:00,204 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:00,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:00,205 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:00,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:00,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:00,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:00,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:00,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:00,208 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:00,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:00,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:00,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:00,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:00,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:00,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:00,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:00,208 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:00,208 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:00,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:00,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:00,208 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:00,308 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:00,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:00,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:00,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:00,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:00,312 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:00,312 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:00,312 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:00,312 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:00,312 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:00,313 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:00,316 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:00,316 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:00,316 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,316 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:00,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:00,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:00,317 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:00,317 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:00,317 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:00,317 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:00,317 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:00,317 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:00,317 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:00,317 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:00,318 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:56:00,318 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-26 13:56:00,318 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:56:00,318 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:00,318 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:56:00,318 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:00,318 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:56:00,318 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:00,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:56:00,550 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:56:00,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:00,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839020566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:00,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839020570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:00,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839020571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:00,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839020572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:00,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839020573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:00,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839020676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:00,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839020676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:00,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839020678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:00,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839020678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,726 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/b0dc37e9dcbf41dbaffb19b8b0856ff5 2023-03-26 13:56:00,734 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/92770713c98b4f1189951d93dbab8ed7 2023-03-26 13:56:00,741 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/3b0295ea993745c9bbb32fd775611344 2023-03-26 13:56:00,744 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/b0dc37e9dcbf41dbaffb19b8b0856ff5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b0dc37e9dcbf41dbaffb19b8b0856ff5 2023-03-26 13:56:00,747 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b0dc37e9dcbf41dbaffb19b8b0856ff5, entries=150, sequenceid=608, filesize=11.9 K 2023-03-26 13:56:00,747 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/92770713c98b4f1189951d93dbab8ed7 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/92770713c98b4f1189951d93dbab8ed7 2023-03-26 13:56:00,750 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/92770713c98b4f1189951d93dbab8ed7, entries=150, sequenceid=608, filesize=11.9 K 2023-03-26 13:56:00,751 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/3b0295ea993745c9bbb32fd775611344 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/3b0295ea993745c9bbb32fd775611344 2023-03-26 13:56:00,753 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/3b0295ea993745c9bbb32fd775611344, entries=150, sequenceid=608, filesize=11.9 K 2023-03-26 13:56:00,754 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2990228151772e454b24e80da60e1bb2 in 436ms, sequenceid=608, compaction requested=true 2023-03-26 13:56:00,754 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:56:00,754 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:00,754 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:00,754 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:00,754 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,754 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:00,754 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:00,755 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:00,755 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:00,756 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:00,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:00,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fb06d92[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:00,757 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:00,757 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:00,758 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:00,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:00,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:00,759 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:00,759 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:00,759 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:00,759 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,759 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:00,760 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,760 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:00,760 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:00,760 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:00,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:00,762 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:00,762 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:00,762 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:00,762 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:00,762 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:00,767 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:00,767 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:00,767 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:00,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:00,767 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:00,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:00,768 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:00,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:00,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,770 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:00,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:00,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:00,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:00,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:00,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:00,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:00,771 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:00,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:00,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:00,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:00,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:00,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:00,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:00,771 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:00,772 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:00,872 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:00,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:00,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:00,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:00,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:00,875 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:00,875 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:00,875 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:00,875 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:00,875 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:00,875 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:00,876 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:00,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:00,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:00,876 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:00,877 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,877 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:00,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:00,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:00,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:00,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:00,878 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:00,878 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:00,878 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:00,878 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:00,878 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:00,878 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:00,878 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:56:00,878 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-26 13:56:00,878 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:56:00,878 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:00,878 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:56:00,878 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:00,878 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:56:00,878 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:00,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:56:00,884 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:56:00,908 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:00,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839020902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:00,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839020902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:00,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839020903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:00,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:00,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839020905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:01,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:01,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839021009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:01,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:01,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839021010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:01,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:01,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839021011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:01,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:01,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839021011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:01,214 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:01,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839021211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:01,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:01,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839021216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:01,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:01,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839021216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:01,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:01,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839021217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:01,286 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/bdf5ed72963941629b599451f06c73e8 2023-03-26 13:56:01,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:01,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839021516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:01,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:01,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839021523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:01,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:01,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839021523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:01,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:01,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839021526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:01,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:01,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59310 deadline: 1679839021579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:01,701 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/5196a0cec863420fa1fb22fae65df57f 2023-03-26 13:56:02,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:02,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839022021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:02,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839022027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:02,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839022029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,037 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:02,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839022035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,110 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=625 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/d2e4d12eac544e40aa0472bc1bafb802 2023-03-26 13:56:02,113 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/bdf5ed72963941629b599451f06c73e8 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/bdf5ed72963941629b599451f06c73e8 2023-03-26 13:56:02,117 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/bdf5ed72963941629b599451f06c73e8, entries=150, sequenceid=625, filesize=11.9 K 2023-03-26 13:56:02,117 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/5196a0cec863420fa1fb22fae65df57f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/5196a0cec863420fa1fb22fae65df57f 2023-03-26 13:56:02,121 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/5196a0cec863420fa1fb22fae65df57f, entries=150, sequenceid=625, filesize=11.9 K 2023-03-26 13:56:02,122 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/d2e4d12eac544e40aa0472bc1bafb802 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/d2e4d12eac544e40aa0472bc1bafb802 2023-03-26 13:56:02,130 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/d2e4d12eac544e40aa0472bc1bafb802, entries=150, sequenceid=625, filesize=11.9 K 2023-03-26 13:56:02,131 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2990228151772e454b24e80da60e1bb2 in 1253ms, sequenceid=625, compaction requested=true 2023-03-26 13:56:02,131 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:56:02,131 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:02,131 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:02,131 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:02,131 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,131 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:02,131 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:02,133 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,133 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:02,133 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:02,133 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:02,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:02,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:02,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@480cf73b[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:02,135 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:02,136 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:02,138 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:02,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:02,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:02,138 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:02,138 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:02,138 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:02,138 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,138 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:02,140 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,140 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:02,140 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:02,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:02,140 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:02,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:02,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:02,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:02,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:02,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:02,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:02,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:02,146 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:02,146 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:02,146 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:02,146 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:02,146 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:02,148 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:02,148 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:02,148 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:02,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:02,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:02,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:02,151 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,153 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,153 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:02,153 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:02,153 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,154 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:02,154 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:02,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:02,154 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:02,155 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:02,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:02,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:02,155 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:02,155 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:02,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:02,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:02,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:02,155 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:02,155 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:02,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:02,255 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:02,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:02,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:02,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:02,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:02,259 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:02,259 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:02,259 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:02,259 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:02,259 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:02,259 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:02,261 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:02,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:02,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:02,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:02,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:02,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:02,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:02,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:02,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:02,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:02,263 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:02,263 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:02,263 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:02,263 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:02,263 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:02,263 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:02,263 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:56:02,263 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-26 13:56:02,263 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:56:02,264 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:02,264 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:56:02,264 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:02,264 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:56:02,264 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:02,274 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/9fb44b1d470649d695f03b73e46d1f0b 2023-03-26 13:56:02,689 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/06f508a38bd247e682b4724c1c16a968 2023-03-26 13:56:03,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:56:03,034 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:56:03,099 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/063a72e28f364fd29ecf73f3335644c2 2023-03-26 13:56:03,105 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/9fb44b1d470649d695f03b73e46d1f0b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/9fb44b1d470649d695f03b73e46d1f0b 2023-03-26 13:56:03,109 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/9fb44b1d470649d695f03b73e46d1f0b, entries=150, sequenceid=640, filesize=11.9 K 2023-03-26 13:56:03,110 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/06f508a38bd247e682b4724c1c16a968 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/06f508a38bd247e682b4724c1c16a968 2023-03-26 13:56:03,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:03,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839023093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,114 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/06f508a38bd247e682b4724c1c16a968, entries=150, sequenceid=640, filesize=11.9 K 2023-03-26 13:56:03,116 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/063a72e28f364fd29ecf73f3335644c2 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/063a72e28f364fd29ecf73f3335644c2 2023-03-26 13:56:03,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:03,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839023105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,123 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:03,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839023110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:03,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839023112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,127 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/063a72e28f364fd29ecf73f3335644c2, entries=150, sequenceid=640, filesize=11.9 K 2023-03-26 13:56:03,128 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2990228151772e454b24e80da60e1bb2 in 865ms, sequenceid=640, compaction requested=true 2023-03-26 13:56:03,128 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:56:03,128 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:03,128 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:03,128 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:03,128 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,132 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:03,132 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:03,133 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,133 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:03,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:03,134 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,134 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:03,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:03,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:03,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:03,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:03,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:03,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79fcd958[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:03,135 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:03,136 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,137 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:03,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:03,137 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:03,138 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,138 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:03,139 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:03,139 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:03,139 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:03,139 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:03,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:03,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:03,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:03,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:03,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:03,142 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:03,142 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:03,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:03,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:03,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:03,142 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:03,145 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,145 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:03,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,145 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:03,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,146 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:03,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,147 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:03,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:03,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,149 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,149 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:03,149 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:03,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:03,149 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:03,150 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,150 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,150 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,150 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:03,150 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:03,150 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:03,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:03,150 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:03,150 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:03,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:03,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:03,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:56:03,221 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-26 13:56:03,222 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:56:03,222 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:03,222 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:56:03,222 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:03,222 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:56:03,222 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:03,250 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/c9c4ab63cb7e41c0926bcea3dc47ff9f 2023-03-26 13:56:03,250 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:03,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:03,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:03,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:03,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:03,260 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:03,260 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:03,260 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:03,260 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:03,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,261 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:03,262 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:03,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:03,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:03,262 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:03,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:03,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:03,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:03,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:03,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:03,264 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:03,264 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:03,264 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:03,264 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:03,264 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:03,264 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:56:03,264 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:03,264 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:56:03,264 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:03,264 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:03,264 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:03,264 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,269 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:03,269 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:03,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:03,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839023255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,270 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,270 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:03,271 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,271 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:03,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:03,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:03,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839023257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:03,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839023258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:03,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:03,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f4e4a4c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:03,274 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:03,274 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,276 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,277 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:03,277 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:03,277 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:03,277 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,277 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:03,278 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/f0ad2ceed1e64e9dbc8ef2e4aaaa699b 2023-03-26 13:56:03,279 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:03,279 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:03,279 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,279 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:03,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:03,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:03,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:03,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:03,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:03,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:03,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:03,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:03,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:03,282 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:03,282 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:03,282 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:03,282 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:03,282 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:03,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:03,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839023269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,284 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,284 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:03,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,284 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:03,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:03,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:03,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:03,286 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:03,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,290 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:03,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:03,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,298 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,298 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:03,298 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:03,298 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,298 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,298 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,298 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,298 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:03,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:03,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:03,298 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:03,298 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:03,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:03,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:03,298 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:03,298 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:03,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:03,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:03,322 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=658 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/99478ca69fee445086ff7d8f8100b96e 2023-03-26 13:56:03,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/c9c4ab63cb7e41c0926bcea3dc47ff9f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/c9c4ab63cb7e41c0926bcea3dc47ff9f 2023-03-26 13:56:03,332 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/c9c4ab63cb7e41c0926bcea3dc47ff9f, entries=200, sequenceid=658, filesize=14.3 K 2023-03-26 13:56:03,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/f0ad2ceed1e64e9dbc8ef2e4aaaa699b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/f0ad2ceed1e64e9dbc8ef2e4aaaa699b 2023-03-26 13:56:03,350 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/f0ad2ceed1e64e9dbc8ef2e4aaaa699b, entries=150, sequenceid=658, filesize=11.9 K 2023-03-26 13:56:03,350 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/99478ca69fee445086ff7d8f8100b96e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/99478ca69fee445086ff7d8f8100b96e 2023-03-26 13:56:03,354 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/99478ca69fee445086ff7d8f8100b96e, entries=150, sequenceid=658, filesize=11.9 K 2023-03-26 13:56:03,355 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 2990228151772e454b24e80da60e1bb2 in 134ms, sequenceid=658, compaction requested=true 2023-03-26 13:56:03,355 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:56:03,355 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:03,355 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:03,355 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 13:56:03,355 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 13:56:03,355 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-26 13:56:03,357 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 76884 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 13:56:03,357 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/A is initiating minor compaction (all files) 2023-03-26 13:56:03,357 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/A in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:03,357 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/c0da6d26846b4856888091961d4142e9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/f4bd265f3e4345c78131647fcc563fbd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b0dc37e9dcbf41dbaffb19b8b0856ff5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/bdf5ed72963941629b599451f06c73e8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/9fb44b1d470649d695f03b73e46d1f0b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/c9c4ab63cb7e41c0926bcea3dc47ff9f] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=75.1 K 2023-03-26 13:56:03,357 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74444 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 13:56:03,358 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/B is initiating minor compaction (all files) 2023-03-26 13:56:03,358 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/B in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:03,358 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/3b5484503f694d878381e3b971b8c012, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a1ca560748ad4b86b0424c22ab2652ad, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/92770713c98b4f1189951d93dbab8ed7, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/5196a0cec863420fa1fb22fae65df57f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/06f508a38bd247e682b4724c1c16a968, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/f0ad2ceed1e64e9dbc8ef2e4aaaa699b] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=72.7 K 2023-03-26 13:56:03,358 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting c0da6d26846b4856888091961d4142e9, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1679838958666 2023-03-26 13:56:03,364 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 3b5484503f694d878381e3b971b8c012, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1679838958666 2023-03-26 13:56:03,364 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting a1ca560748ad4b86b0424c22ab2652ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1679838958792 2023-03-26 13:56:03,365 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting f4bd265f3e4345c78131647fcc563fbd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1679838958792 2023-03-26 13:56:03,365 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 92770713c98b4f1189951d93dbab8ed7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1679838959428 2023-03-26 13:56:03,365 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 5196a0cec863420fa1fb22fae65df57f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1679838960570 2023-03-26 13:56:03,365 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 06f508a38bd247e682b4724c1c16a968, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1679838960884 2023-03-26 13:56:03,365 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting f0ad2ceed1e64e9dbc8ef2e4aaaa699b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1679838963105 2023-03-26 13:56:03,368 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting b0dc37e9dcbf41dbaffb19b8b0856ff5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1679838959428 2023-03-26 13:56:03,372 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting bdf5ed72963941629b599451f06c73e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1679838960570 2023-03-26 13:56:03,372 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 9fb44b1d470649d695f03b73e46d1f0b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1679838960884 2023-03-26 13:56:03,372 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting c9c4ab63cb7e41c0926bcea3dc47ff9f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1679838963049 2023-03-26 13:56:03,381 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-26 13:56:03,382 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:56:03,382 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:03,382 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:56:03,382 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:03,382 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:56:03,382 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:03,382 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#B#compaction#570 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:03,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:56:03,399 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:03,399 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#A#compaction#571 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:03,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:03,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:03,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:03,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:03,433 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:03,433 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:03,433 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:03,433 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:03,435 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/cc1e22fdcf0f4c72a1c357d82af29c24 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/cc1e22fdcf0f4c72a1c357d82af29c24 2023-03-26 13:56:03,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:03,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:03,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839023427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:03,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:03,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:03,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:03,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:03,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:03,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:03,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:03,443 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2990228151772e454b24e80da60e1bb2/B of 2990228151772e454b24e80da60e1bb2 into cc1e22fdcf0f4c72a1c357d82af29c24(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:03,443 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:56:03,443 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/B, priority=10, startTime=1679838963355; duration=0sec 2023-03-26 13:56:03,444 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:03,444 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 13:56:03,445 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74342 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 13:56:03,445 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): 2990228151772e454b24e80da60e1bb2/C is initiating minor compaction (all files) 2023-03-26 13:56:03,445 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2990228151772e454b24e80da60e1bb2/C in TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:03,445 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/57d5682a41014e5eb7601797ba350cfb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/5249d344b90b46fa889e5241e4957de2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/3b0295ea993745c9bbb32fd775611344, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/d2e4d12eac544e40aa0472bc1bafb802, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/063a72e28f364fd29ecf73f3335644c2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/99478ca69fee445086ff7d8f8100b96e] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp, totalSize=72.6 K 2023-03-26 13:56:03,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:03,446 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 57d5682a41014e5eb7601797ba350cfb, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=573, earliestPutTs=1679838958666 2023-03-26 13:56:03,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59340 deadline: 1679839023433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:03,446 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 5249d344b90b46fa889e5241e4957de2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=590, earliestPutTs=1679838958792 2023-03-26 13:56:03,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59286 deadline: 1679839023434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:03,446 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 3b0295ea993745c9bbb32fd775611344, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1679838959428 2023-03-26 13:56:03,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59284 deadline: 1679839023437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,447 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting d2e4d12eac544e40aa0472bc1bafb802, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=625, earliestPutTs=1679838960570 2023-03-26 13:56:03,447 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 063a72e28f364fd29ecf73f3335644c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1679838960884 2023-03-26 13:56:03,448 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 99478ca69fee445086ff7d8f8100b96e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=658, earliestPutTs=1679838963105 2023-03-26 13:56:03,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:03,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:03,456 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:03,456 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:03,456 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:03,459 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/1d460968254343c294942e65b18c5a95 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/1d460968254343c294942e65b18c5a95 2023-03-26 13:56:03,462 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2990228151772e454b24e80da60e1bb2/A of 2990228151772e454b24e80da60e1bb2 into 1d460968254343c294942e65b18c5a95(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:03,463 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:56:03,463 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/A, priority=10, startTime=1679838963355; duration=0sec 2023-03-26 13:56:03,463 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:03,466 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2990228151772e454b24e80da60e1bb2#C#compaction#573 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:03,467 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/3f018b626f1f4e01870d2be042ec8aa5 2023-03-26 13:56:03,472 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:03,472 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:03,472 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:03,472 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. started... 2023-03-26 13:56:03,472 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. as already flushing 2023-03-26 13:56:03,472 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:03,472 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:03,472 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:03,472 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,480 DEBUG [Thread-3415] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24a917df to 127.0.0.1:54332 2023-03-26 13:56:03,480 DEBUG [Thread-3415] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:03,480 DEBUG [Thread-3419] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x36d2d826 to 127.0.0.1:54332 2023-03-26 13:56:03,481 DEBUG [Thread-3419] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:03,483 DEBUG [Thread-3411] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x028a3a77 to 127.0.0.1:54332 2023-03-26 13:56:03,483 DEBUG [Thread-3411] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:03,484 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:03,484 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:03,487 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,487 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:03,487 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,487 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43fe7eca[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:03,489 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:03,489 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,490 DEBUG [Thread-3413] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4cefb012 to 127.0.0.1:54332 2023-03-26 13:56:03,490 DEBUG [Thread-3413] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:03,490 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,490 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:03,490 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:03,490 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:03,491 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,491 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:03,493 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:03,493 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:03,493 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:03,493 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:03,495 DEBUG [Thread-3417] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78fa3dee to 127.0.0.1:54332 2023-03-26 13:56:03,495 DEBUG [Thread-3417] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:03,495 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/478a920bc7cb4d2fad342912bd701806 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/478a920bc7cb4d2fad342912bd701806 2023-03-26 13:56:03,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:03,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:03,498 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/b2f72c057a8c4f3b811b77b4aa6069b8 2023-03-26 13:56:03,498 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:03,498 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:03,498 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:03,498 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:03,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:03,498 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:03,500 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2990228151772e454b24e80da60e1bb2/C of 2990228151772e454b24e80da60e1bb2 into 478a920bc7cb4d2fad342912bd701806(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:03,500 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,500 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:03,500 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:03,500 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:56:03,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:03,501 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:03,500 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2., storeName=2990228151772e454b24e80da60e1bb2/C, priority=10, startTime=1679838963355; duration=0sec 2023-03-26 13:56:03,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:03,501 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:03,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,502 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:03,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:03,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,504 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,504 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:03,504 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:03,504 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,504 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,504 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:03,504 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:03,504 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:03,504 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:03,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:03,504 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:03,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:03,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:03,505 DEBUG [Thread-3410] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:03,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:03,505 DEBUG [Thread-3410] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:03,528 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/ff6dc2c54587482290f5b2b443fe0ad0 2023-03-26 13:56:03,531 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/3f018b626f1f4e01870d2be042ec8aa5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/3f018b626f1f4e01870d2be042ec8aa5 2023-03-26 13:56:03,535 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/3f018b626f1f4e01870d2be042ec8aa5, entries=200, sequenceid=673, filesize=14.3 K 2023-03-26 13:56:03,535 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/b2f72c057a8c4f3b811b77b4aa6069b8 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/b2f72c057a8c4f3b811b77b4aa6069b8 2023-03-26 13:56:03,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:03,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:59354 deadline: 1679839023538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,541 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/b2f72c057a8c4f3b811b77b4aa6069b8, entries=150, sequenceid=673, filesize=11.9 K 2023-03-26 13:56:03,542 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/ff6dc2c54587482290f5b2b443fe0ad0 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ff6dc2c54587482290f5b2b443fe0ad0 2023-03-26 13:56:03,545 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ff6dc2c54587482290f5b2b443fe0ad0, entries=150, sequenceid=673, filesize=11.9 K 2023-03-26 13:56:03,546 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 2990228151772e454b24e80da60e1bb2 in 165ms, sequenceid=673, compaction requested=false 2023-03-26 13:56:03,546 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:56:03,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on 2990228151772e454b24e80da60e1bb2 2023-03-26 13:56:03,549 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-26 13:56:03,550 DEBUG [Thread-3406] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x32cd672d to 127.0.0.1:54332 2023-03-26 13:56:03,550 DEBUG [Thread-3400] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3fb41c50 to 127.0.0.1:54332 2023-03-26 13:56:03,550 DEBUG [Thread-3400] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:03,550 DEBUG [Thread-3406] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:03,551 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:56:03,551 DEBUG [Thread-3402] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x357cb237 to 127.0.0.1:54332 2023-03-26 13:56:03,551 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:03,551 DEBUG [Thread-3402] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:03,551 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:56:03,552 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:03,552 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:56:03,552 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:03,565 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=696 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/888139b5be664f05b95d657c3c6f6474 2023-03-26 13:56:03,600 DEBUG [Thread-3408] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x09b8ff30 to 127.0.0.1:54332 2023-03-26 13:56:03,600 DEBUG [Thread-3408] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:03,605 DEBUG [Thread-3410] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:03,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:03,742 DEBUG [Thread-3404] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4539ed0c to 127.0.0.1:54332 2023-03-26 13:56:03,742 DEBUG [Thread-3404] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:03,742 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-26 13:56:03,743 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 99 2023-03-26 13:56:03,743 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 119 2023-03-26 13:56:03,743 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 110 2023-03-26 13:56:03,743 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 117 2023-03-26 13:56:03,743 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 106 2023-03-26 13:56:03,743 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-26 13:56:03,743 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-26 13:56:03,743 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(402): scanned 2212 2023-03-26 13:56:03,743 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(403): verified 6630 rows 2023-03-26 13:56:03,743 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(402): scanned 2231 2023-03-26 13:56:03,743 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(403): verified 6690 rows 2023-03-26 13:56:03,743 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(402): scanned 2217 2023-03-26 13:56:03,743 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(403): verified 6648 rows 2023-03-26 13:56:03,743 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(402): scanned 2222 2023-03-26 13:56:03,744 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(403): verified 6663 rows 2023-03-26 13:56:03,744 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(402): scanned 2232 2023-03-26 13:56:03,744 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(403): verified 6693 rows 2023-03-26 13:56:03,744 INFO [Listener at localhost/41531] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-26 13:56:03,744 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2c8fbc35 to 127.0.0.1:54332 2023-03-26 13:56:03,744 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:03,747 INFO [Listener at localhost/41531] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-26 13:56:03,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-26 13:56:03,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-26 13:56:03,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-26 13:56:03,753 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838963753"}]},"ts":"1679838963753"} 2023-03-26 13:56:03,754 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-26 13:56:03,849 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-26 13:56:03,850 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2990228151772e454b24e80da60e1bb2, UNASSIGN}] 2023-03-26 13:56:03,852 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2990228151772e454b24e80da60e1bb2, UNASSIGN 2023-03-26 13:56:03,852 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=2990228151772e454b24e80da60e1bb2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:03,853 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838963852"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838963852"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838963852"}]},"ts":"1679838963852"} 2023-03-26 13:56:03,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-26 13:56:03,854 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:56:03,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=696 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/ec6ee5b5ebc14c9aba1b2d85c9baac75 2023-03-26 13:56:04,005 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2990228151772e454b24e80da60e1bb2 2023-03-26 13:56:04,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2990228151772e454b24e80da60e1bb2, disabling compactions & flushes 2023-03-26 13:56:04,006 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:04,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-26 13:56:04,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-26 13:56:04,392 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=696 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/7fcad99a7ee34c308c91b6b648d82177 2023-03-26 13:56:04,395 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/888139b5be664f05b95d657c3c6f6474 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/888139b5be664f05b95d657c3c6f6474 2023-03-26 13:56:04,398 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/888139b5be664f05b95d657c3c6f6474, entries=150, sequenceid=696, filesize=11.9 K 2023-03-26 13:56:04,399 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/ec6ee5b5ebc14c9aba1b2d85c9baac75 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/ec6ee5b5ebc14c9aba1b2d85c9baac75 2023-03-26 13:56:04,402 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/ec6ee5b5ebc14c9aba1b2d85c9baac75, entries=150, sequenceid=696, filesize=11.9 K 2023-03-26 13:56:04,402 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/7fcad99a7ee34c308c91b6b648d82177 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/7fcad99a7ee34c308c91b6b648d82177 2023-03-26 13:56:04,405 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/7fcad99a7ee34c308c91b6b648d82177, entries=150, sequenceid=696, filesize=11.9 K 2023-03-26 13:56:04,406 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=13.42 KB/13740 for 2990228151772e454b24e80da60e1bb2 in 857ms, sequenceid=696, compaction requested=true 2023-03-26 13:56:04,406 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:56:04,406 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:04,406 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. because compaction request was cancelled 2023-03-26 13:56:04,406 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:04,406 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:04,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. after waiting 0 ms 2023-03-26 13:56:04,407 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. because compaction request was cancelled 2023-03-26 13:56:04,407 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:04,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:04,407 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. because compaction request was cancelled 2023-03-26 13:56:04,407 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 2990228151772e454b24e80da60e1bb2 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-03-26 13:56:04,407 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:04,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=A 2023-03-26 13:56:04,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:04,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=B 2023-03-26 13:56:04,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:04,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2990228151772e454b24e80da60e1bb2, store=C 2023-03-26 13:56:04,407 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:04,813 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/3284a6b505924939856c0741faec87e5 2023-03-26 13:56:04,822 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/9b021dd233b146369484026700fbd8e1 2023-03-26 13:56:04,831 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=701 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/f4dceeded8a247bd9f1a48c952937fbc 2023-03-26 13:56:04,835 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/A/3284a6b505924939856c0741faec87e5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/3284a6b505924939856c0741faec87e5 2023-03-26 13:56:04,838 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/3284a6b505924939856c0741faec87e5, entries=100, sequenceid=701, filesize=9.5 K 2023-03-26 13:56:04,838 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/B/9b021dd233b146369484026700fbd8e1 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9b021dd233b146369484026700fbd8e1 2023-03-26 13:56:04,842 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9b021dd233b146369484026700fbd8e1, entries=100, sequenceid=701, filesize=9.5 K 2023-03-26 13:56:04,842 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/.tmp/C/f4dceeded8a247bd9f1a48c952937fbc as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/f4dceeded8a247bd9f1a48c952937fbc 2023-03-26 13:56:04,845 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/f4dceeded8a247bd9f1a48c952937fbc, entries=100, sequenceid=701, filesize=9.5 K 2023-03-26 13:56:04,846 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for 2990228151772e454b24e80da60e1bb2 in 439ms, sequenceid=701, compaction requested=true 2023-03-26 13:56:04,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-26 13:56:04,880 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/155b936dc9c54663b8d709c2e00a1629, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/0f5457e1beca41bcab9d6580cf2fec2f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/e5114cfa21574166b7a9403bb617f46f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/4446a5c2924b44b3a61a50facf090991, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/7e698084699a4dee914a9b48e83a5cb0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/ac47ae6c228042dbb042a684ec91d555, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/dfcef5bee5344e3793df5f5f741ba2fe, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/d199f1349acc4b7abf4b0e135df92383, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/97221351266e4bc181fcfbddab811523, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/a7f7ce591e3b49a1b63620485392bcd4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b079d71a53a743e38b2f618c42199844, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/66fbc2cf0b4f42b2a286a48bca001115, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/eebd0bf91a3d42fb905a2e3fa4eb3ad5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/d727767162634314ab67f7fb0affa56e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/7b6ccc3103da409b8e376cd1d38c82ac, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/49fb33e7f1034d10ab91850b02b55422, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/51a5ffda71ce49339fc158ca8f0f31a1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/2c9dd99d59df4fe3973f1494be6d65fc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/ff219c2625db4c1f9e7845bd4f5eb1cd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/54f35ccc19e24274acdee1fa62c5fc93, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/cb0edf2d5ed9489e8cda52dac1d69c9d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b46ec9ef145b4c098a4120c61805d9b1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/040ce1b4fc94475491a6c15902639ac9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/3490784ced664c1cad277fe3a50694ae, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/29744925d4554a3fa1293980bccf49c5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/25baed7286ca4185b2bdb478d063f3f4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/52464a85a7c240a885959214c72a8364, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/4cbc3880fba546b3845e6c8756d6fc27, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/f05690db23ef4f7c9c9999096747a45b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/c1dd2ff904d74b14a8b0d6c36bd5c285, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/7ebff05f3a68430fbb3d80218e65be09, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/1733113733cc43339e8e190e45f8edfd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/aac3822f7dc64578848b4c7163c9d3c8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b26009bb60a84890b8daec590d213b5b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/ed508833cd7547099b648686af8e45b7, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/f9b1f18fa7324236ba6f266cd409df5e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/2be6be788f764b7ea91870a4c929299d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/13ca310abcd246ffa8deffab998dcb5a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/caba37a5685548db9eedf4ed87bbc18d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/c0da6d26846b4856888091961d4142e9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/f4bd265f3e4345c78131647fcc563fbd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b0dc37e9dcbf41dbaffb19b8b0856ff5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/bdf5ed72963941629b599451f06c73e8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/9fb44b1d470649d695f03b73e46d1f0b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/c9c4ab63cb7e41c0926bcea3dc47ff9f] to archive 2023-03-26 13:56:04,881 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 13:56:04,882 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/155b936dc9c54663b8d709c2e00a1629 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/155b936dc9c54663b8d709c2e00a1629 2023-03-26 13:56:04,884 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/0f5457e1beca41bcab9d6580cf2fec2f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/0f5457e1beca41bcab9d6580cf2fec2f 2023-03-26 13:56:04,885 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/e5114cfa21574166b7a9403bb617f46f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/e5114cfa21574166b7a9403bb617f46f 2023-03-26 13:56:04,886 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/4446a5c2924b44b3a61a50facf090991 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/4446a5c2924b44b3a61a50facf090991 2023-03-26 13:56:04,887 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/7e698084699a4dee914a9b48e83a5cb0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/7e698084699a4dee914a9b48e83a5cb0 2023-03-26 13:56:04,888 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/ac47ae6c228042dbb042a684ec91d555 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/ac47ae6c228042dbb042a684ec91d555 2023-03-26 13:56:04,889 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/dfcef5bee5344e3793df5f5f741ba2fe to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/dfcef5bee5344e3793df5f5f741ba2fe 2023-03-26 13:56:04,890 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/d199f1349acc4b7abf4b0e135df92383 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/d199f1349acc4b7abf4b0e135df92383 2023-03-26 13:56:04,891 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/97221351266e4bc181fcfbddab811523 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/97221351266e4bc181fcfbddab811523 2023-03-26 13:56:04,892 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/a7f7ce591e3b49a1b63620485392bcd4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/a7f7ce591e3b49a1b63620485392bcd4 2023-03-26 13:56:04,893 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b079d71a53a743e38b2f618c42199844 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b079d71a53a743e38b2f618c42199844 2023-03-26 13:56:04,894 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/66fbc2cf0b4f42b2a286a48bca001115 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/66fbc2cf0b4f42b2a286a48bca001115 2023-03-26 13:56:04,895 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/eebd0bf91a3d42fb905a2e3fa4eb3ad5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/eebd0bf91a3d42fb905a2e3fa4eb3ad5 2023-03-26 13:56:04,896 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/d727767162634314ab67f7fb0affa56e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/d727767162634314ab67f7fb0affa56e 2023-03-26 13:56:04,897 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/7b6ccc3103da409b8e376cd1d38c82ac to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/7b6ccc3103da409b8e376cd1d38c82ac 2023-03-26 13:56:04,899 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/49fb33e7f1034d10ab91850b02b55422 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/49fb33e7f1034d10ab91850b02b55422 2023-03-26 13:56:04,900 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/51a5ffda71ce49339fc158ca8f0f31a1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/51a5ffda71ce49339fc158ca8f0f31a1 2023-03-26 13:56:04,901 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/2c9dd99d59df4fe3973f1494be6d65fc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/2c9dd99d59df4fe3973f1494be6d65fc 2023-03-26 13:56:04,902 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/ff219c2625db4c1f9e7845bd4f5eb1cd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/ff219c2625db4c1f9e7845bd4f5eb1cd 2023-03-26 13:56:04,904 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/54f35ccc19e24274acdee1fa62c5fc93 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/54f35ccc19e24274acdee1fa62c5fc93 2023-03-26 13:56:04,905 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/cb0edf2d5ed9489e8cda52dac1d69c9d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/cb0edf2d5ed9489e8cda52dac1d69c9d 2023-03-26 13:56:04,907 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b46ec9ef145b4c098a4120c61805d9b1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b46ec9ef145b4c098a4120c61805d9b1 2023-03-26 13:56:04,908 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/040ce1b4fc94475491a6c15902639ac9 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/040ce1b4fc94475491a6c15902639ac9 2023-03-26 13:56:04,909 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/3490784ced664c1cad277fe3a50694ae to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/3490784ced664c1cad277fe3a50694ae 2023-03-26 13:56:04,910 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/29744925d4554a3fa1293980bccf49c5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/29744925d4554a3fa1293980bccf49c5 2023-03-26 13:56:04,911 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/25baed7286ca4185b2bdb478d063f3f4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/25baed7286ca4185b2bdb478d063f3f4 2023-03-26 13:56:04,912 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/52464a85a7c240a885959214c72a8364 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/52464a85a7c240a885959214c72a8364 2023-03-26 13:56:04,914 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/4cbc3880fba546b3845e6c8756d6fc27 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/4cbc3880fba546b3845e6c8756d6fc27 2023-03-26 13:56:04,915 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/f05690db23ef4f7c9c9999096747a45b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/f05690db23ef4f7c9c9999096747a45b 2023-03-26 13:56:04,916 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/c1dd2ff904d74b14a8b0d6c36bd5c285 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/c1dd2ff904d74b14a8b0d6c36bd5c285 2023-03-26 13:56:04,918 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/7ebff05f3a68430fbb3d80218e65be09 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/7ebff05f3a68430fbb3d80218e65be09 2023-03-26 13:56:04,919 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/1733113733cc43339e8e190e45f8edfd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/1733113733cc43339e8e190e45f8edfd 2023-03-26 13:56:04,920 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/aac3822f7dc64578848b4c7163c9d3c8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/aac3822f7dc64578848b4c7163c9d3c8 2023-03-26 13:56:04,921 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b26009bb60a84890b8daec590d213b5b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b26009bb60a84890b8daec590d213b5b 2023-03-26 13:56:04,922 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/ed508833cd7547099b648686af8e45b7 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/ed508833cd7547099b648686af8e45b7 2023-03-26 13:56:04,923 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/f9b1f18fa7324236ba6f266cd409df5e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/f9b1f18fa7324236ba6f266cd409df5e 2023-03-26 13:56:04,924 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/2be6be788f764b7ea91870a4c929299d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/2be6be788f764b7ea91870a4c929299d 2023-03-26 13:56:04,925 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/13ca310abcd246ffa8deffab998dcb5a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/13ca310abcd246ffa8deffab998dcb5a 2023-03-26 13:56:04,926 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/caba37a5685548db9eedf4ed87bbc18d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/caba37a5685548db9eedf4ed87bbc18d 2023-03-26 13:56:04,927 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/c0da6d26846b4856888091961d4142e9 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/c0da6d26846b4856888091961d4142e9 2023-03-26 13:56:04,928 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/f4bd265f3e4345c78131647fcc563fbd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/f4bd265f3e4345c78131647fcc563fbd 2023-03-26 13:56:04,929 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b0dc37e9dcbf41dbaffb19b8b0856ff5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/b0dc37e9dcbf41dbaffb19b8b0856ff5 2023-03-26 13:56:04,930 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/bdf5ed72963941629b599451f06c73e8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/bdf5ed72963941629b599451f06c73e8 2023-03-26 13:56:04,931 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/9fb44b1d470649d695f03b73e46d1f0b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/9fb44b1d470649d695f03b73e46d1f0b 2023-03-26 13:56:04,932 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/c9c4ab63cb7e41c0926bcea3dc47ff9f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/c9c4ab63cb7e41c0926bcea3dc47ff9f 2023-03-26 13:56:04,936 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/42ba44fa7021473fb680205385fe5bdf, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/bca19ce9987a4300977b8c34d9a96fda, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/e0317c78fe0e416ea1f7399746d978ce, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/4deca1641fa44b5c9fc9fdb8f4ecf00e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/12525b1709b34a3b874ceadece43e338, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/d8bdfb87dcde472c818669b1029cd136, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/d0a6dac1b87e4811a826f14db0f5dea7, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/10b9c1d3fd864c678aa79ee3d794131b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/743f7c3699b444a3875d6c0d1a0970e5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/01833fa91b44470ab67fdc9f1aa2c551, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9d6a2b188a6f4f52a46d83f464a67f0d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/e05205493a1943ceb60f5a1c1b291791, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/2c092c7a972941038ce0a6b3692b9603, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/db268056546f4b4d96fe3502dc512a2d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/19be7a99b67c4cb283b2208717809e1b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/88d3d5248b884ff7a7e75a9cb37a8323, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9a86e8fa9a20488e8152851a2643c1b5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/7f8097a9f4834835a266a76071d9920e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/33b5fb4caa5743cca9f3bc2040e652c9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/901f631fa9b4467c99cee436875e420c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9e1ef49fa8424a95818a5d8708bf723d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a2abeaf8a96c4416a1ae558b23766d0d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/1d44e350151d4641bbe8815c96f927c7, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/8b4586b602584320a8af8e3b0a988e98, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/33478942a7414c39a3330d403c0911de, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/7a255090fee943379934d749a4b46316, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a720ae392ac14e519394b64413549435, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a8208a538dcf4e54be4ef576f8de31e9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/f8860e3621684134a40aaa8ecdb3fea0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/ab638aef9d7f4e6782985accd53b2e11, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/fdf6c9953c814283bad56cc1fbfcef37, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/39f6865fa7d146018a395c851495fab9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a077e121521b42cf99b199e7716a4f54, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/52fdf3c4a4b44b1bbe4c26903bb9f7b4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/0710c44047544d44befa93f9e0197309, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/088fa57a1d054400a8711ac2e55f4c14, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/08f9e2dae5084281a2c739bfaacfdfbe, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/94efc9a58fe94742844b8ff16935b07c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/3b5484503f694d878381e3b971b8c012, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9484019c4e4043e3b535ed74deca8e99, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a1ca560748ad4b86b0424c22ab2652ad, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/92770713c98b4f1189951d93dbab8ed7, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/5196a0cec863420fa1fb22fae65df57f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/06f508a38bd247e682b4724c1c16a968, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/f0ad2ceed1e64e9dbc8ef2e4aaaa699b] to archive 2023-03-26 13:56:04,937 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 13:56:04,939 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/42ba44fa7021473fb680205385fe5bdf to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/42ba44fa7021473fb680205385fe5bdf 2023-03-26 13:56:04,940 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/bca19ce9987a4300977b8c34d9a96fda to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/bca19ce9987a4300977b8c34d9a96fda 2023-03-26 13:56:04,941 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/e0317c78fe0e416ea1f7399746d978ce to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/e0317c78fe0e416ea1f7399746d978ce 2023-03-26 13:56:04,942 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/4deca1641fa44b5c9fc9fdb8f4ecf00e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/4deca1641fa44b5c9fc9fdb8f4ecf00e 2023-03-26 13:56:04,943 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/12525b1709b34a3b874ceadece43e338 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/12525b1709b34a3b874ceadece43e338 2023-03-26 13:56:04,944 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/d8bdfb87dcde472c818669b1029cd136 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/d8bdfb87dcde472c818669b1029cd136 2023-03-26 13:56:04,945 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/d0a6dac1b87e4811a826f14db0f5dea7 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/d0a6dac1b87e4811a826f14db0f5dea7 2023-03-26 13:56:04,946 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/10b9c1d3fd864c678aa79ee3d794131b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/10b9c1d3fd864c678aa79ee3d794131b 2023-03-26 13:56:04,947 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/743f7c3699b444a3875d6c0d1a0970e5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/743f7c3699b444a3875d6c0d1a0970e5 2023-03-26 13:56:04,948 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/01833fa91b44470ab67fdc9f1aa2c551 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/01833fa91b44470ab67fdc9f1aa2c551 2023-03-26 13:56:04,948 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9d6a2b188a6f4f52a46d83f464a67f0d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9d6a2b188a6f4f52a46d83f464a67f0d 2023-03-26 13:56:04,950 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/e05205493a1943ceb60f5a1c1b291791 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/e05205493a1943ceb60f5a1c1b291791 2023-03-26 13:56:04,951 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/2c092c7a972941038ce0a6b3692b9603 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/2c092c7a972941038ce0a6b3692b9603 2023-03-26 13:56:04,952 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/db268056546f4b4d96fe3502dc512a2d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/db268056546f4b4d96fe3502dc512a2d 2023-03-26 13:56:04,953 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/19be7a99b67c4cb283b2208717809e1b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/19be7a99b67c4cb283b2208717809e1b 2023-03-26 13:56:04,954 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/88d3d5248b884ff7a7e75a9cb37a8323 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/88d3d5248b884ff7a7e75a9cb37a8323 2023-03-26 13:56:04,955 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9a86e8fa9a20488e8152851a2643c1b5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9a86e8fa9a20488e8152851a2643c1b5 2023-03-26 13:56:04,956 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/7f8097a9f4834835a266a76071d9920e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/7f8097a9f4834835a266a76071d9920e 2023-03-26 13:56:04,957 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/33b5fb4caa5743cca9f3bc2040e652c9 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/33b5fb4caa5743cca9f3bc2040e652c9 2023-03-26 13:56:04,958 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/901f631fa9b4467c99cee436875e420c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/901f631fa9b4467c99cee436875e420c 2023-03-26 13:56:04,959 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9e1ef49fa8424a95818a5d8708bf723d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9e1ef49fa8424a95818a5d8708bf723d 2023-03-26 13:56:04,961 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a2abeaf8a96c4416a1ae558b23766d0d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a2abeaf8a96c4416a1ae558b23766d0d 2023-03-26 13:56:04,962 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/1d44e350151d4641bbe8815c96f927c7 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/1d44e350151d4641bbe8815c96f927c7 2023-03-26 13:56:04,963 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/8b4586b602584320a8af8e3b0a988e98 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/8b4586b602584320a8af8e3b0a988e98 2023-03-26 13:56:04,964 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/33478942a7414c39a3330d403c0911de to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/33478942a7414c39a3330d403c0911de 2023-03-26 13:56:04,965 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/7a255090fee943379934d749a4b46316 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/7a255090fee943379934d749a4b46316 2023-03-26 13:56:04,966 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a720ae392ac14e519394b64413549435 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a720ae392ac14e519394b64413549435 2023-03-26 13:56:04,968 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a8208a538dcf4e54be4ef576f8de31e9 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a8208a538dcf4e54be4ef576f8de31e9 2023-03-26 13:56:04,969 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/f8860e3621684134a40aaa8ecdb3fea0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/f8860e3621684134a40aaa8ecdb3fea0 2023-03-26 13:56:04,970 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/ab638aef9d7f4e6782985accd53b2e11 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/ab638aef9d7f4e6782985accd53b2e11 2023-03-26 13:56:04,971 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/fdf6c9953c814283bad56cc1fbfcef37 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/fdf6c9953c814283bad56cc1fbfcef37 2023-03-26 13:56:04,973 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/39f6865fa7d146018a395c851495fab9 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/39f6865fa7d146018a395c851495fab9 2023-03-26 13:56:04,974 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a077e121521b42cf99b199e7716a4f54 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a077e121521b42cf99b199e7716a4f54 2023-03-26 13:56:04,975 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/52fdf3c4a4b44b1bbe4c26903bb9f7b4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/52fdf3c4a4b44b1bbe4c26903bb9f7b4 2023-03-26 13:56:04,976 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/0710c44047544d44befa93f9e0197309 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/0710c44047544d44befa93f9e0197309 2023-03-26 13:56:04,977 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/088fa57a1d054400a8711ac2e55f4c14 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/088fa57a1d054400a8711ac2e55f4c14 2023-03-26 13:56:04,978 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/08f9e2dae5084281a2c739bfaacfdfbe to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/08f9e2dae5084281a2c739bfaacfdfbe 2023-03-26 13:56:04,979 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/94efc9a58fe94742844b8ff16935b07c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/94efc9a58fe94742844b8ff16935b07c 2023-03-26 13:56:04,980 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/3b5484503f694d878381e3b971b8c012 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/3b5484503f694d878381e3b971b8c012 2023-03-26 13:56:04,982 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9484019c4e4043e3b535ed74deca8e99 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9484019c4e4043e3b535ed74deca8e99 2023-03-26 13:56:04,983 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a1ca560748ad4b86b0424c22ab2652ad to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/a1ca560748ad4b86b0424c22ab2652ad 2023-03-26 13:56:04,984 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/92770713c98b4f1189951d93dbab8ed7 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/92770713c98b4f1189951d93dbab8ed7 2023-03-26 13:56:04,985 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/5196a0cec863420fa1fb22fae65df57f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/5196a0cec863420fa1fb22fae65df57f 2023-03-26 13:56:04,986 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/06f508a38bd247e682b4724c1c16a968 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/06f508a38bd247e682b4724c1c16a968 2023-03-26 13:56:04,987 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/f0ad2ceed1e64e9dbc8ef2e4aaaa699b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/f0ad2ceed1e64e9dbc8ef2e4aaaa699b 2023-03-26 13:56:04,991 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/6b03de4565424ceca121924c02578f81, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/3cd68ae5d8254b20975a41788f0eb484, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c2531883b9434d058a61d0fe91716054, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c0e3e1b29bf14b999d52f01f0e0529bf, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b3e66f98a3e84b318e66f7f8e0aecdc0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/2d5639a9a1054f04a9fded3472d1b7ce, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/a9ab7df802b24ec0b4058ed9cb3ebd59, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/bf0131b798cf471aa9961ab4f57ad064, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/993aa38b7477426e9c7c6b0284755d3a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b1b9845fc1254fecb7713f7b5476283c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/6a45cf1284004427afbbd3b143f5248d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c788d45f7f4344649f7b01404cc60af1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c518bd17bfaf41f784ab95b083e09baf, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b04e3dc6d46949b793e1a8a9d37bb931, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/8e04ee371b9a49baba1f4ee2f6299b3d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/bb49e162b9ff49c9952bdc1d88a0c854, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/1da4573419274ad29a3f54dc49152257, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ea32489048714164ac2921f7b942c8cc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/a7980d2aad564ea783316c95329670d1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c4d6ab73383644019e1fac8d42ab0aa7, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/130c02e135f44720b673ab5372d5463e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/0c04ff44194a4ab1bbca7cb6c9f092c2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/9d8ae4309eb64ee9adc328baa44c7ee7, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b73231c61e38487388a9437c3706a52c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/8aade3946dd04b8b8605e7f23dce51d0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/5def744979d94ed5bf446bdb4001d395, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/6083ba607b094299b0710904de1a0100, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/4d070fe488bd4020870217d55e309132, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/2d38473ba1b14282b095930c63ef945a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ea8ef92dc9e74f51acd8e5f896f8e8e1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ee03866624884317a0a1b77a04a03adc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/4eb053d4a2bd4b5daf4d8041d3671783, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/aff090b84a084216811f040d8e0462e3, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/fecf2a19a3ec41b5940bd333e8706325, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b51754b36cf14eb7b88cbb6823e6699c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/57d5682a41014e5eb7601797ba350cfb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/4379db71c232469f88c20f8e2ad22a4b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/5249d344b90b46fa889e5241e4957de2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/3b0295ea993745c9bbb32fd775611344, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/d2e4d12eac544e40aa0472bc1bafb802, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/063a72e28f364fd29ecf73f3335644c2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/99478ca69fee445086ff7d8f8100b96e] to archive 2023-03-26 13:56:04,992 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 13:56:04,993 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/6b03de4565424ceca121924c02578f81 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/6b03de4565424ceca121924c02578f81 2023-03-26 13:56:04,994 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/3cd68ae5d8254b20975a41788f0eb484 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/3cd68ae5d8254b20975a41788f0eb484 2023-03-26 13:56:04,995 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c2531883b9434d058a61d0fe91716054 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c2531883b9434d058a61d0fe91716054 2023-03-26 13:56:04,996 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c0e3e1b29bf14b999d52f01f0e0529bf to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c0e3e1b29bf14b999d52f01f0e0529bf 2023-03-26 13:56:04,997 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b3e66f98a3e84b318e66f7f8e0aecdc0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b3e66f98a3e84b318e66f7f8e0aecdc0 2023-03-26 13:56:04,998 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/2d5639a9a1054f04a9fded3472d1b7ce to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/2d5639a9a1054f04a9fded3472d1b7ce 2023-03-26 13:56:04,999 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/a9ab7df802b24ec0b4058ed9cb3ebd59 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/a9ab7df802b24ec0b4058ed9cb3ebd59 2023-03-26 13:56:05,000 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/bf0131b798cf471aa9961ab4f57ad064 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/bf0131b798cf471aa9961ab4f57ad064 2023-03-26 13:56:05,001 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/993aa38b7477426e9c7c6b0284755d3a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/993aa38b7477426e9c7c6b0284755d3a 2023-03-26 13:56:05,001 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b1b9845fc1254fecb7713f7b5476283c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b1b9845fc1254fecb7713f7b5476283c 2023-03-26 13:56:05,002 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/6a45cf1284004427afbbd3b143f5248d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/6a45cf1284004427afbbd3b143f5248d 2023-03-26 13:56:05,003 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c788d45f7f4344649f7b01404cc60af1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c788d45f7f4344649f7b01404cc60af1 2023-03-26 13:56:05,004 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c518bd17bfaf41f784ab95b083e09baf to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c518bd17bfaf41f784ab95b083e09baf 2023-03-26 13:56:05,005 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b04e3dc6d46949b793e1a8a9d37bb931 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b04e3dc6d46949b793e1a8a9d37bb931 2023-03-26 13:56:05,006 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/8e04ee371b9a49baba1f4ee2f6299b3d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/8e04ee371b9a49baba1f4ee2f6299b3d 2023-03-26 13:56:05,008 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/bb49e162b9ff49c9952bdc1d88a0c854 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/bb49e162b9ff49c9952bdc1d88a0c854 2023-03-26 13:56:05,009 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/1da4573419274ad29a3f54dc49152257 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/1da4573419274ad29a3f54dc49152257 2023-03-26 13:56:05,010 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ea32489048714164ac2921f7b942c8cc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ea32489048714164ac2921f7b942c8cc 2023-03-26 13:56:05,011 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/a7980d2aad564ea783316c95329670d1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/a7980d2aad564ea783316c95329670d1 2023-03-26 13:56:05,012 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c4d6ab73383644019e1fac8d42ab0aa7 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/c4d6ab73383644019e1fac8d42ab0aa7 2023-03-26 13:56:05,021 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/130c02e135f44720b673ab5372d5463e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/130c02e135f44720b673ab5372d5463e 2023-03-26 13:56:05,023 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/0c04ff44194a4ab1bbca7cb6c9f092c2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/0c04ff44194a4ab1bbca7cb6c9f092c2 2023-03-26 13:56:05,024 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/9d8ae4309eb64ee9adc328baa44c7ee7 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/9d8ae4309eb64ee9adc328baa44c7ee7 2023-03-26 13:56:05,025 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b73231c61e38487388a9437c3706a52c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b73231c61e38487388a9437c3706a52c 2023-03-26 13:56:05,026 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/8aade3946dd04b8b8605e7f23dce51d0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/8aade3946dd04b8b8605e7f23dce51d0 2023-03-26 13:56:05,027 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/5def744979d94ed5bf446bdb4001d395 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/5def744979d94ed5bf446bdb4001d395 2023-03-26 13:56:05,028 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/6083ba607b094299b0710904de1a0100 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/6083ba607b094299b0710904de1a0100 2023-03-26 13:56:05,029 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/4d070fe488bd4020870217d55e309132 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/4d070fe488bd4020870217d55e309132 2023-03-26 13:56:05,030 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/2d38473ba1b14282b095930c63ef945a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/2d38473ba1b14282b095930c63ef945a 2023-03-26 13:56:05,031 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ea8ef92dc9e74f51acd8e5f896f8e8e1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ea8ef92dc9e74f51acd8e5f896f8e8e1 2023-03-26 13:56:05,032 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ee03866624884317a0a1b77a04a03adc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ee03866624884317a0a1b77a04a03adc 2023-03-26 13:56:05,033 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/4eb053d4a2bd4b5daf4d8041d3671783 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/4eb053d4a2bd4b5daf4d8041d3671783 2023-03-26 13:56:05,034 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/aff090b84a084216811f040d8e0462e3 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/aff090b84a084216811f040d8e0462e3 2023-03-26 13:56:05,035 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/fecf2a19a3ec41b5940bd333e8706325 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/fecf2a19a3ec41b5940bd333e8706325 2023-03-26 13:56:05,036 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b51754b36cf14eb7b88cbb6823e6699c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/b51754b36cf14eb7b88cbb6823e6699c 2023-03-26 13:56:05,037 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/57d5682a41014e5eb7601797ba350cfb to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/57d5682a41014e5eb7601797ba350cfb 2023-03-26 13:56:05,038 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/4379db71c232469f88c20f8e2ad22a4b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/4379db71c232469f88c20f8e2ad22a4b 2023-03-26 13:56:05,040 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/5249d344b90b46fa889e5241e4957de2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/5249d344b90b46fa889e5241e4957de2 2023-03-26 13:56:05,041 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/3b0295ea993745c9bbb32fd775611344 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/3b0295ea993745c9bbb32fd775611344 2023-03-26 13:56:05,042 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/d2e4d12eac544e40aa0472bc1bafb802 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/d2e4d12eac544e40aa0472bc1bafb802 2023-03-26 13:56:05,043 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/063a72e28f364fd29ecf73f3335644c2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/063a72e28f364fd29ecf73f3335644c2 2023-03-26 13:56:05,045 DEBUG [StoreCloser-TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/99478ca69fee445086ff7d8f8100b96e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/99478ca69fee445086ff7d8f8100b96e 2023-03-26 13:56:05,051 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/recovered.edits/704.seqid, newMaxSeqId=704, maxSeqId=1 2023-03-26 13:56:05,053 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2. 2023-03-26 13:56:05,053 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2990228151772e454b24e80da60e1bb2: 2023-03-26 13:56:05,054 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2990228151772e454b24e80da60e1bb2 2023-03-26 13:56:05,055 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=2990228151772e454b24e80da60e1bb2, regionState=CLOSED 2023-03-26 13:56:05,055 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838965055"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679838965055"}]},"ts":"1679838965055"} 2023-03-26 13:56:05,057 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-03-26 13:56:05,057 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 2990228151772e454b24e80da60e1bb2, server=jenkins-hbase4.apache.org,42517,1679838814721 in 1.2020 sec 2023-03-26 13:56:05,058 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-03-26 13:56:05,059 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2990228151772e454b24e80da60e1bb2, UNASSIGN in 1.2070 sec 2023-03-26 13:56:05,059 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838965059"}]},"ts":"1679838965059"} 2023-03-26 13:56:05,060 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-26 13:56:05,061 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-26 13:56:05,063 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.3140 sec 2023-03-26 13:56:05,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-26 13:56:05,868 INFO [Listener at localhost/41531] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-03-26 13:56:05,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-26 13:56:05,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:56:05,871 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:56:05,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-26 13:56:05,872 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:56:05,878 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2 2023-03-26 13:56:05,881 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A, FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B, FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C, FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/recovered.edits] 2023-03-26 13:56:05,884 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/1d460968254343c294942e65b18c5a95 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/1d460968254343c294942e65b18c5a95 2023-03-26 13:56:05,885 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/3284a6b505924939856c0741faec87e5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/3284a6b505924939856c0741faec87e5 2023-03-26 13:56:05,886 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/3f018b626f1f4e01870d2be042ec8aa5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/3f018b626f1f4e01870d2be042ec8aa5 2023-03-26 13:56:05,887 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/888139b5be664f05b95d657c3c6f6474 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/A/888139b5be664f05b95d657c3c6f6474 2023-03-26 13:56:05,890 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9b021dd233b146369484026700fbd8e1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/9b021dd233b146369484026700fbd8e1 2023-03-26 13:56:05,891 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/b2f72c057a8c4f3b811b77b4aa6069b8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/b2f72c057a8c4f3b811b77b4aa6069b8 2023-03-26 13:56:05,892 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/cc1e22fdcf0f4c72a1c357d82af29c24 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/cc1e22fdcf0f4c72a1c357d82af29c24 2023-03-26 13:56:05,893 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/ec6ee5b5ebc14c9aba1b2d85c9baac75 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/B/ec6ee5b5ebc14c9aba1b2d85c9baac75 2023-03-26 13:56:05,895 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/478a920bc7cb4d2fad342912bd701806 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/478a920bc7cb4d2fad342912bd701806 2023-03-26 13:56:05,896 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/7fcad99a7ee34c308c91b6b648d82177 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/7fcad99a7ee34c308c91b6b648d82177 2023-03-26 13:56:05,897 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/f4dceeded8a247bd9f1a48c952937fbc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/f4dceeded8a247bd9f1a48c952937fbc 2023-03-26 13:56:05,898 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ff6dc2c54587482290f5b2b443fe0ad0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/C/ff6dc2c54587482290f5b2b443fe0ad0 2023-03-26 13:56:05,900 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/recovered.edits/704.seqid to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2/recovered.edits/704.seqid 2023-03-26 13:56:05,901 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/2990228151772e454b24e80da60e1bb2 2023-03-26 13:56:05,901 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 13:56:05,902 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:56:05,904 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-26 13:56:05,905 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-26 13:56:05,906 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:56:05,906 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-26 13:56:05,906 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679838965906"}]},"ts":"9223372036854775807"} 2023-03-26 13:56:05,907 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-26 13:56:05,907 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2990228151772e454b24e80da60e1bb2, NAME => 'TestAcidGuarantees,,1679838942309.2990228151772e454b24e80da60e1bb2.', STARTKEY => '', ENDKEY => ''}] 2023-03-26 13:56:05,907 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-26 13:56:05,907 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679838965907"}]},"ts":"9223372036854775807"} 2023-03-26 13:56:05,909 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-26 13:56:05,911 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:56:05,911 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 42 msec 2023-03-26 13:56:05,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-26 13:56:05,973 INFO [Listener at localhost/41531] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-03-26 13:56:05,988 INFO [Listener at localhost/41531] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=235 (was 227) - Thread LEAK? -, OpenFileDescriptor=608 (was 510) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=412 (was 334) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=9009 (was 8619) - AvailableMemoryMB LEAK? - 2023-03-26 13:56:06,002 INFO [Listener at localhost/41531] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=235, OpenFileDescriptor=608, MaxFileDescriptor=60000, SystemLoadAverage=412, ProcessCount=171, AvailableMemoryMB=8996 2023-03-26 13:56:06,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-26 13:56:06,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 13:56:06,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-26 13:56:06,007 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-26 13:56:06,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-03-26 13:56:06,008 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-26 13:56:06,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-26 13:56:06,009 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:06,009 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048 empty. 2023-03-26 13:56:06,010 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:06,010 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 13:56:06,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-26 13:56:06,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-26 13:56:06,416 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-26 13:56:06,417 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => f904470a09c7c6a52bdab844746de048, NAME => 'TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp 2023-03-26 13:56:06,427 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:56:06,427 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing f904470a09c7c6a52bdab844746de048, disabling compactions & flushes 2023-03-26 13:56:06,427 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:06,427 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:06,427 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. after waiting 0 ms 2023-03-26 13:56:06,427 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:06,427 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:06,427 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:06,429 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-26 13:56:06,430 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838966430"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679838966430"}]},"ts":"1679838966430"} 2023-03-26 13:56:06,431 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-26 13:56:06,432 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-26 13:56:06,432 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838966432"}]},"ts":"1679838966432"} 2023-03-26 13:56:06,433 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-26 13:56:06,437 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f904470a09c7c6a52bdab844746de048, ASSIGN}] 2023-03-26 13:56:06,438 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f904470a09c7c6a52bdab844746de048, ASSIGN 2023-03-26 13:56:06,439 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=f904470a09c7c6a52bdab844746de048, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,42517,1679838814721; forceNewPlan=false, retain=false 2023-03-26 13:56:06,590 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=f904470a09c7c6a52bdab844746de048, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:06,590 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838966590"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838966590"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838966590"}]},"ts":"1679838966590"} 2023-03-26 13:56:06,592 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:56:06,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-26 13:56:06,746 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:06,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f904470a09c7c6a52bdab844746de048, NAME => 'TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.', STARTKEY => '', ENDKEY => ''} 2023-03-26 13:56:06,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:06,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:56:06,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:06,747 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:06,748 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:06,749 DEBUG [StoreOpener-f904470a09c7c6a52bdab844746de048-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A 2023-03-26 13:56:06,749 DEBUG [StoreOpener-f904470a09c7c6a52bdab844746de048-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A 2023-03-26 13:56:06,749 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:56:06,750 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f904470a09c7c6a52bdab844746de048 columnFamilyName A 2023-03-26 13:56:06,750 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] regionserver.HStore(310): Store=f904470a09c7c6a52bdab844746de048/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:56:06,750 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:06,751 DEBUG [StoreOpener-f904470a09c7c6a52bdab844746de048-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B 2023-03-26 13:56:06,751 DEBUG [StoreOpener-f904470a09c7c6a52bdab844746de048-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B 2023-03-26 13:56:06,751 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:56:06,751 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f904470a09c7c6a52bdab844746de048 columnFamilyName B 2023-03-26 13:56:06,752 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] regionserver.HStore(310): Store=f904470a09c7c6a52bdab844746de048/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:56:06,752 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:06,753 DEBUG [StoreOpener-f904470a09c7c6a52bdab844746de048-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C 2023-03-26 13:56:06,753 DEBUG [StoreOpener-f904470a09c7c6a52bdab844746de048-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C 2023-03-26 13:56:06,753 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:56:06,753 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f904470a09c7c6a52bdab844746de048 columnFamilyName C 2023-03-26 13:56:06,754 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] regionserver.HStore(310): Store=f904470a09c7c6a52bdab844746de048/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:56:06,754 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:06,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:06,755 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:06,757 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 13:56:06,772 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:06,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-26 13:56:06,774 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f904470a09c7c6a52bdab844746de048; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71172184, jitterRate=0.06054818630218506}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 13:56:06,774 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:06,775 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., pid=56, masterSystemTime=1679838966743 2023-03-26 13:56:06,776 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:06,776 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:06,776 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=f904470a09c7c6a52bdab844746de048, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:06,777 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838966776"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679838966776"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679838966776"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679838966776"}]},"ts":"1679838966776"} 2023-03-26 13:56:06,779 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-03-26 13:56:06,779 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 in 185 msec 2023-03-26 13:56:06,780 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-03-26 13:56:06,780 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f904470a09c7c6a52bdab844746de048, ASSIGN in 342 msec 2023-03-26 13:56:06,780 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-26 13:56:06,780 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838966780"}]},"ts":"1679838966780"} 2023-03-26 13:56:06,781 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-26 13:56:06,783 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-26 13:56:06,784 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 778 msec 2023-03-26 13:56:07,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-26 13:56:07,112 INFO [Listener at localhost/41531] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-03-26 13:56:07,113 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x55d0f313 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:56:07,117 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d207bb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:56:07,118 DEBUG [Listener at localhost/41531] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:56:07,120 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46328, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:56:07,121 DEBUG [Listener at localhost/41531] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-26 13:56:07,123 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59678, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-26 13:56:07,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-26 13:56:07,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 13:56:07,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-26 13:56:07,538 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-26 13:56:07,539 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-26 13:56:07,539 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-26 13:56:07,541 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-26 13:56:07,543 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f904470a09c7c6a52bdab844746de048, REOPEN/MOVE}] 2023-03-26 13:56:07,544 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f904470a09c7c6a52bdab844746de048, REOPEN/MOVE 2023-03-26 13:56:07,544 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=f904470a09c7c6a52bdab844746de048, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:07,544 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838967544"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838967544"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838967544"}]},"ts":"1679838967544"} 2023-03-26 13:56:07,545 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:56:07,697 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:07,698 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f904470a09c7c6a52bdab844746de048, disabling compactions & flushes 2023-03-26 13:56:07,698 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:07,698 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:07,698 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. after waiting 0 ms 2023-03-26 13:56:07,698 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:07,703 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-26 13:56:07,704 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:07,704 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:07,704 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: f904470a09c7c6a52bdab844746de048 to self. 2023-03-26 13:56:07,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:07,706 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=f904470a09c7c6a52bdab844746de048, regionState=CLOSED 2023-03-26 13:56:07,706 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838967706"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679838967706"}]},"ts":"1679838967706"} 2023-03-26 13:56:07,709 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-03-26 13:56:07,709 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 in 162 msec 2023-03-26 13:56:07,709 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=f904470a09c7c6a52bdab844746de048, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,42517,1679838814721; forceNewPlan=false, retain=true 2023-03-26 13:56:07,860 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=f904470a09c7c6a52bdab844746de048, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:07,860 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838967860"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838967860"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838967860"}]},"ts":"1679838967860"} 2023-03-26 13:56:07,861 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:56:08,015 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:08,016 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f904470a09c7c6a52bdab844746de048, NAME => 'TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.', STARTKEY => '', ENDKEY => ''} 2023-03-26 13:56:08,016 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:08,016 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 13:56:08,016 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:08,016 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:08,017 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:08,018 DEBUG [StoreOpener-f904470a09c7c6a52bdab844746de048-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A 2023-03-26 13:56:08,018 DEBUG [StoreOpener-f904470a09c7c6a52bdab844746de048-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A 2023-03-26 13:56:08,018 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:56:08,019 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f904470a09c7c6a52bdab844746de048 columnFamilyName A 2023-03-26 13:56:08,020 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] regionserver.HStore(310): Store=f904470a09c7c6a52bdab844746de048/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:56:08,020 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:08,021 DEBUG [StoreOpener-f904470a09c7c6a52bdab844746de048-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B 2023-03-26 13:56:08,021 DEBUG [StoreOpener-f904470a09c7c6a52bdab844746de048-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B 2023-03-26 13:56:08,021 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:56:08,021 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f904470a09c7c6a52bdab844746de048 columnFamilyName B 2023-03-26 13:56:08,021 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] regionserver.HStore(310): Store=f904470a09c7c6a52bdab844746de048/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:56:08,021 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:08,022 DEBUG [StoreOpener-f904470a09c7c6a52bdab844746de048-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C 2023-03-26 13:56:08,022 DEBUG [StoreOpener-f904470a09c7c6a52bdab844746de048-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C 2023-03-26 13:56:08,022 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 13:56:08,022 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f904470a09c7c6a52bdab844746de048 columnFamilyName C 2023-03-26 13:56:08,023 INFO [StoreOpener-f904470a09c7c6a52bdab844746de048-1] regionserver.HStore(310): Store=f904470a09c7c6a52bdab844746de048/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 13:56:08,023 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:08,024 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:08,025 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:08,026 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 13:56:08,027 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:08,028 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened f904470a09c7c6a52bdab844746de048; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72767907, jitterRate=0.08432631194591522}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 13:56:08,028 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:08,029 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., pid=61, masterSystemTime=1679838968013 2023-03-26 13:56:08,030 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:08,030 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:08,030 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=f904470a09c7c6a52bdab844746de048, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,030 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838968030"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679838968030"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679838968030"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679838968030"}]},"ts":"1679838968030"} 2023-03-26 13:56:08,032 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-03-26 13:56:08,032 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 in 170 msec 2023-03-26 13:56:08,033 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-03-26 13:56:08,033 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f904470a09c7c6a52bdab844746de048, REOPEN/MOVE in 489 msec 2023-03-26 13:56:08,035 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-03-26 13:56:08,035 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 493 msec 2023-03-26 13:56:08,037 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 910 msec 2023-03-26 13:56:08,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-03-26 13:56:08,038 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x4a320817 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:56:08,044 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a4acea2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:56:08,044 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x19c1e3ee to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:56:08,050 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43693271, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:56:08,050 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x526f2221 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:56:08,055 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44277511, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:56:08,055 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x17cf2e3f to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:56:08,059 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c1a68cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:56:08,059 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x3420e54c to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:56:08,065 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a709b21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:56:08,065 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x156e4ba6 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:56:08,070 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4b1973b9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:56:08,071 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x785e366b to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:56:08,079 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f0a71bd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:56:08,079 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x5fdfef3b to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:56:08,089 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56bb77c5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:56:08,089 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x77f9399b to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:56:08,094 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@577fa668, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:56:08,095 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(139): Connect 0x430757a6 to 127.0.0.1:54332 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 13:56:08,103 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d8c8bbc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 13:56:08,112 DEBUG [hconnection-0x31e649e6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:56:08,117 DEBUG [hconnection-0x6fe8431e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:56:08,118 DEBUG [hconnection-0x46ead174-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:56:08,119 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46350, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:56:08,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:08,121 DEBUG [hconnection-0x1ee6ba16-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:56:08,121 DEBUG [hconnection-0x21e881b5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:56:08,122 DEBUG [hconnection-0x5d3d58ba-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:56:08,123 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46384, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:56:08,123 DEBUG [hconnection-0x4ca818fc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:56:08,124 DEBUG [hconnection-0x78fc390e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:56:08,129 DEBUG [hconnection-0x41ff7af5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:56:08,133 DEBUG [hconnection-0x202b75f9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 13:56:08,134 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46358, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:56:08,134 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46334, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:56:08,134 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46380, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:56:08,134 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46364, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:56:08,134 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46394, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:56:08,134 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46388, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:56:08,136 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46406, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:56:08,138 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46408, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 13:56:08,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:08,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:08,142 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:08,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:08,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:08,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:08,142 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,143 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,146 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:08,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:08,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:08,147 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,148 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,148 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:08,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:08,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:08,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:08,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:08,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:08,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:08,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:08,155 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 13:56:08,155 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:08,155 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:08,155 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:08,155 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:08,155 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:08,155 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:08,155 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:08,155 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:08,155 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,155 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:08,156 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:08,156 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:08,156 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:08,156 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:08,156 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:08,156 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:08,156 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:08,157 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,157 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:08,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:08,157 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,157 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:08,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:08,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:08,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:08,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c64fca1[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:08,173 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:08,173 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,174 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839028170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,174 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,175 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,175 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:08,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839028172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:08,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:08,175 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:08,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839028172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839028173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839028173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,179 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,179 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:08,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:08,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:08,179 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:08,180 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:08,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:08,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:08,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:08,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:08,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:08,183 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:08,183 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:08,183 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:08,183 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:08,183 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:08,186 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,186 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:08,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:08,186 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:08,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:08,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:08,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:08,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,188 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:08,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,189 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:08,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:08,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:08,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,195 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:08,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:08,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:08,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:08,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,195 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:08,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:08,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:08,196 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:08,196 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:08,206 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303263d92ca8bc92b40abaa200a2a6b9e5b96 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263d92ca8bc92b40abaa200a2a6b9e5b96 2023-03-26 13:56:08,219 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=17, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/90c2a16f4af84d9db1e0bb2bb45a144c 2023-03-26 13:56:08,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839028276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839028276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839028276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839028276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839028277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,296 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:08,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:08,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:08,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:08,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:08,299 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:08,299 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:08,299 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:08,300 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:08,300 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,300 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,302 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:08,302 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:08,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:08,303 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,303 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:08,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:08,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:08,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:08,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:08,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:08,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:08,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:08,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:08,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:08,304 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:08,305 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:08,305 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:08,305 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:08,305 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:08,305 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:08,305 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,305 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:08,305 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:08,306 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,306 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:08,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:08,307 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,307 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:08,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:08,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:08,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:08,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:08,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73cdce6c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:08,309 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:08,309 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,310 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,310 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:08,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:08,310 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:08,310 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:08,311 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:08,311 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:08,312 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:08,312 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:08,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:08,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:08,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:08,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:08,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:08,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:08,315 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:08,315 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:08,315 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:08,315 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:08,315 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:08,318 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,318 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,318 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:08,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:08,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:08,318 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:08,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:08,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:08,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:08,320 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:08,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,323 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:08,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:08,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:08,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:08,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:08,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:08,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:08,323 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:08,323 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:08,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:08,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:08,324 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:08,424 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:08,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:08,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:08,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:08,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:08,427 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:08,427 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:08,427 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:08,427 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:08,427 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,427 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,428 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:08,428 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:08,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:08,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,429 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,429 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:08,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:08,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:08,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:08,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:08,430 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:08,430 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:08,430 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:08,430 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:08,430 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:08,430 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:08,431 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:08,431 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:08,431 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:08,431 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:08,431 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:08,431 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,431 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:08,431 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:08,432 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,432 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:08,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:08,432 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,432 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:08,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:08,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:08,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:08,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:08,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e0cf673[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:08,435 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:08,435 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,438 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:08,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:08,438 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:08,438 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:08,439 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:08,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:08,439 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:08,439 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:08,439 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:08,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:08,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:08,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:08,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:08,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:08,442 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:08,442 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:08,442 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:08,443 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:08,443 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:08,445 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,445 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:08,445 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:08,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:08,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:08,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:08,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:08,446 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:08,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,448 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:08,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,450 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:08,451 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:08,451 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:08,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:08,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:08,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:08,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:08,451 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,451 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,451 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,451 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,451 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,451 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,451 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,451 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:08,451 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:08,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:08,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:08,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:08,479 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839028479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839028479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,480 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839028479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839028479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839028480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,551 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:08,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:08,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:08,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:08,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:08,560 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:08,560 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:08,560 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:08,560 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:08,560 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,560 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,563 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:08,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:08,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:08,563 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,563 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,564 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:08,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:08,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:08,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:08,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:08,565 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:08,565 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:08,565 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:08,568 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:08,568 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:08,568 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:08,569 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:08,569 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:08,569 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:08,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:08,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:08,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:08,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:08,570 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:08,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:08,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:08,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:08,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:08,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:08,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:08,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e34c556[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:08,573 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:08,573 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,574 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,574 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:08,574 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:08,574 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:08,574 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,575 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:08,576 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,576 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:08,576 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:08,576 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:08,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:08,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:08,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:08,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:08,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:08,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:08,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:08,579 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:08,579 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:08,579 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:08,579 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:08,579 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:08,582 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,582 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:08,582 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:08,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:08,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:08,583 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:08,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:08,584 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:08,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:08,589 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:08,589 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:08,589 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:08,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:08,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:08,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:08,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:08,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:08,589 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,590 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,590 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,590 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,590 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:08,590 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:08,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:08,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:08,590 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,590 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:08,590 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,638 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/1e7265ad98b34235a81b53dce03ed87d 2023-03-26 13:56:08,659 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=17 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/1d82424a4ddf466787044d6ba1570b6a 2023-03-26 13:56:08,665 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/90c2a16f4af84d9db1e0bb2bb45a144c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/90c2a16f4af84d9db1e0bb2bb45a144c 2023-03-26 13:56:08,670 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/90c2a16f4af84d9db1e0bb2bb45a144c, entries=150, sequenceid=17, filesize=25.1 K 2023-03-26 13:56:08,671 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/1e7265ad98b34235a81b53dce03ed87d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1e7265ad98b34235a81b53dce03ed87d 2023-03-26 13:56:08,674 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1e7265ad98b34235a81b53dce03ed87d, entries=150, sequenceid=17, filesize=11.6 K 2023-03-26 13:56:08,675 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/1d82424a4ddf466787044d6ba1570b6a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/1d82424a4ddf466787044d6ba1570b6a 2023-03-26 13:56:08,679 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/1d82424a4ddf466787044d6ba1570b6a, entries=150, sequenceid=17, filesize=11.6 K 2023-03-26 13:56:08,679 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for f904470a09c7c6a52bdab844746de048 in 525ms, sequenceid=17, compaction requested=false 2023-03-26 13:56:08,679 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-26 13:56:08,680 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:08,690 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:08,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:08,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:08,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:08,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:08,695 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:08,695 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:08,695 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:08,695 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:08,696 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,696 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,697 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:08,697 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:08,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:08,698 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,698 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:08,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:08,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:08,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:08,699 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:08,699 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:08,699 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:08,699 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:08,699 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:08,699 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:08,700 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:08,700 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:08,700 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:08,700 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:08,700 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-26 13:56:08,700 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:08,701 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:08,701 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:08,701 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:08,701 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:08,701 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:08,726 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303261a801abe30a34f718ea82384f577a592 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303261a801abe30a34f718ea82384f577a592 2023-03-26 13:56:08,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:08,782 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:08,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839028787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839028787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839028787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839028789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839028789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839028891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839028891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839028892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839028892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:08,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:08,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839028892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839029095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839029095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839029096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839029096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839029096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,139 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/8ac1b81b14e04108a8537bbc0811eb22 2023-03-26 13:56:09,151 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/1e2b333f69e64de48dd0c3180bdcc576 2023-03-26 13:56:09,164 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/02cddce57c2b4004bda9351afc72f0da 2023-03-26 13:56:09,171 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/8ac1b81b14e04108a8537bbc0811eb22 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/8ac1b81b14e04108a8537bbc0811eb22 2023-03-26 13:56:09,176 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/8ac1b81b14e04108a8537bbc0811eb22, entries=150, sequenceid=37, filesize=25.1 K 2023-03-26 13:56:09,177 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/1e2b333f69e64de48dd0c3180bdcc576 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1e2b333f69e64de48dd0c3180bdcc576 2023-03-26 13:56:09,182 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1e2b333f69e64de48dd0c3180bdcc576, entries=150, sequenceid=37, filesize=11.6 K 2023-03-26 13:56:09,183 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/02cddce57c2b4004bda9351afc72f0da as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/02cddce57c2b4004bda9351afc72f0da 2023-03-26 13:56:09,191 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/02cddce57c2b4004bda9351afc72f0da, entries=150, sequenceid=37, filesize=11.6 K 2023-03-26 13:56:09,192 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for f904470a09c7c6a52bdab844746de048 in 492ms, sequenceid=37, compaction requested=false 2023-03-26 13:56:09,193 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:09,193 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:09,193 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:09,193 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:09,193 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,194 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:09,194 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:09,196 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:09,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:09,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:09,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:09,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:09,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:09,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:09,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@557ea3c5[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:09,199 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:09,199 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,200 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,200 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:09,200 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:09,200 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:09,204 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,204 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:09,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,208 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:09,208 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:09,209 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:09,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:09,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:09,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:09,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:09,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:09,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:09,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:09,212 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:09,212 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:09,213 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:09,213 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:09,213 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:09,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:09,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:09,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:09,215 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:09,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:09,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:09,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:09,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,216 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,220 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,220 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,220 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:09,220 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:09,220 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:09,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:09,220 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,220 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,220 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,220 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,220 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:09,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:09,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:09,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:09,220 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:09,221 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:09,321 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:09,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:09,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:09,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:09,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:09,324 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:09,324 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:09,324 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:09,324 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:09,325 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,325 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,326 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:09,326 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:09,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:09,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,326 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,327 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:09,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:09,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:09,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:09,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:09,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:09,328 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:09,328 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:09,328 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:09,328 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:09,328 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:09,328 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:09,328 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 13:56:09,328 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:09,328 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:09,328 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:09,328 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:09,328 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:09,329 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:09,366 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326a47259ff82ea44aab43e772fed276603 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a47259ff82ea44aab43e772fed276603 2023-03-26 13:56:09,378 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=49, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/0ad62a672be04218ba8805695786ad7a 2023-03-26 13:56:09,390 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/433907e55f86483582fa4b23cfc249ee 2023-03-26 13:56:09,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:09,402 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:09,409 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=49 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/b20f49074f5049cfa424062eddd93c35 2023-03-26 13:56:09,415 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/0ad62a672be04218ba8805695786ad7a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/0ad62a672be04218ba8805695786ad7a 2023-03-26 13:56:09,419 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/0ad62a672be04218ba8805695786ad7a, entries=150, sequenceid=49, filesize=25.1 K 2023-03-26 13:56:09,419 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/433907e55f86483582fa4b23cfc249ee as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/433907e55f86483582fa4b23cfc249ee 2023-03-26 13:56:09,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839029417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679838969412, max timestamp=1679838969419Num uniques 859; 2023-03-26 13:56:09,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-26 13:56:09,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679838969412, max timestamp=1679838969419Num uniques 859; 2023-03-26 13:56:09,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-26 13:56:09,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679838969412, max timestamp=1679838969419Num uniques 859; 2023-03-26 13:56:09,423 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:09,426 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/433907e55f86483582fa4b23cfc249ee, entries=150, sequenceid=49, filesize=11.6 K 2023-03-26 13:56:09,426 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/b20f49074f5049cfa424062eddd93c35 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/b20f49074f5049cfa424062eddd93c35 2023-03-26 13:56:09,430 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/b20f49074f5049cfa424062eddd93c35, entries=150, sequenceid=49, filesize=11.6 K 2023-03-26 13:56:09,431 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=41.70 KB/42700 for f904470a09c7c6a52bdab844746de048 in 102ms, sequenceid=49, compaction requested=true 2023-03-26 13:56:09,431 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:09,431 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:09,431 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:09,431 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:09,431 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:09,431 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=61.83 KB heapSize=169.09 KB 2023-03-26 13:56:09,431 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:09,431 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:09,431 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:09,431 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:09,431 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:09,431 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:09,431 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:09,431 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:09,433 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:09,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:09,433 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:09,434 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,434 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:09,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:09,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:09,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:09,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48ddb9a8[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:09,441 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:09,441 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,443 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,443 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:09,443 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:09,443 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:09,445 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,445 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:09,447 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:09,447 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:09,447 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:09,447 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:09,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:09,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:09,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:09,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:09,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:09,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:09,452 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:09,452 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:09,452 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:09,452 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:09,452 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:09,454 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303261b8de2364e1249639eed2ea356f9a01e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303261b8de2364e1249639eed2ea356f9a01e 2023-03-26 13:56:09,454 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,454 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:09,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:09,455 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:09,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:09,456 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:09,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,458 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:09,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,460 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,460 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,460 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,460 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,460 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,460 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,460 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:09,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:09,460 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:09,461 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:09,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:09,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:09,460 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:09,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:09,461 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:09,461 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:09,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:09,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:09,464 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=75, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/0993abfab80b410cbed406edc1a56926 2023-03-26 13:56:09,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838969459, max timestamp=1679838969463Num uniques 859; 2023-03-26 13:56:09,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838969459, max timestamp=1679838969463Num uniques 859; 2023-03-26 13:56:09,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:09,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:09,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838969459, max timestamp=1679838969463Num uniques 859; 2023-03-26 13:56:09,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:09,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839029483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839029486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,492 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839029487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839029488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839029522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,561 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:09,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:09,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:09,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:09,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:09,566 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:09,566 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:09,566 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:09,566 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:09,566 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,566 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,568 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:09,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:09,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:09,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,569 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,569 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,569 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:09,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:09,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:09,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:09,574 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:09,574 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:09,574 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:09,584 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:09,584 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:09,587 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:09,587 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:09,587 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:09,587 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:09,587 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:09,587 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:09,587 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,588 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:09,588 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:09,591 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,591 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:09,591 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,591 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:09,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:09,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:09,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:09,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:09,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ab61ae[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:09,593 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:09,593 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839029590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,594 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839029590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,594 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:09,594 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:09,594 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:09,595 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,595 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:09,596 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:09,596 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:09,596 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:09,596 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:09,596 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:09,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839029594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:09,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:09,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:09,598 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:09,598 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:09,598 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:09,598 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:09,599 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:09,599 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839029595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,600 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,600 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:09,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:09,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,600 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:09,601 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:09,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:09,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:09,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:09,608 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:09,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:09,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:09,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:09,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:09,612 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:09,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:09,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:09,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,612 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,612 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:09,613 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:09,713 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:09,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:09,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:09,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:09,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:09,716 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:09,717 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:09,717 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:09,717 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:09,717 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,717 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,721 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,721 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:09,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:09,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:09,722 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,722 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:09,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:09,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,723 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:09,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:09,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:09,723 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:09,723 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:09,723 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:09,723 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:09,723 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:09,723 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:09,723 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:09,724 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:09,724 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:09,724 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:09,724 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:09,724 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,724 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:09,724 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:09,725 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,725 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:09,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:09,726 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,726 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:09,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:09,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:09,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:09,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:09,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@398d1ab8[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:09,728 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:09,728 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,730 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,731 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:09,731 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,731 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:09,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839029727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,731 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:09,731 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:09,732 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:09,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:09,732 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:09,732 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:09,733 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:09,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:09,735 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:09,735 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:09,735 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:09,735 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:09,736 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:09,737 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:09,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:09,737 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,738 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:09,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:09,739 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:09,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,741 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:09,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,743 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:09,743 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:09,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:09,743 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:09,744 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:09,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:09,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:09,744 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,744 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,744 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,744 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:09,744 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:09,744 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,744 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,744 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:09,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:09,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:09,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839029795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839029796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839029798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:09,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839029801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,844 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:09,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:09,847 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:09,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:09,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:09,848 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:09,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:09,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:09,848 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:09,849 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,849 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,851 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:09,851 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:09,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:09,852 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,852 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:09,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:09,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:09,853 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:09,853 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:09,853 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:09,853 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:09,853 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:09,853 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:09,853 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:09,853 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:09,853 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:09,854 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:09,854 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:09,854 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:09,854 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:09,854 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,854 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:09,854 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:09,855 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,855 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:09,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:09,856 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,856 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:09,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:09,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:09,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:09,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:09,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@194de6c7[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:09,858 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:09,858 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,860 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,860 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:09,860 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:09,860 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:09,860 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,860 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:09,862 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,862 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:09,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:09,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:09,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:09,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:09,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:09,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:09,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:09,866 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:09,866 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:09,866 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:09,866 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:09,866 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:09,868 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,868 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:09,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,868 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:09,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:09,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:09,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:09,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:09,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,869 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,871 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:09,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,873 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:09,873 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:09,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:09,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:09,873 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,873 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,873 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,873 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,873 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:09,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:09,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:09,874 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:09,874 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:09,873 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,874 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,873 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:09,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:09,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:09,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=19.73 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/f128f6749e954bbab4f6564e311d4aad 2023-03-26 13:56:09,974 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:09,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:09,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:09,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:09,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:09,978 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:09,978 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:09,978 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:09,978 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:09,979 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,979 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,980 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:09,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:09,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:09,981 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,981 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,981 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:09,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:09,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:09,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:09,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:09,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:09,982 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:09,982 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:09,982 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:09,982 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:09,982 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:09,983 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:09,983 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:09,983 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:09,983 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:09,983 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:09,983 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:09,983 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,983 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:09,983 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:09,985 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,985 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:09,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:09,985 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,985 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:09,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:09,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:09,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:09,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:09,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cf80cbf[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:09,989 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:09,989 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,991 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:09,991 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:09,991 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:09,991 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:09,992 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:09,992 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:09,992 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,993 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:09,993 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:09,993 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:09,993 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:09,993 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:09,993 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:09,993 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:09,993 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:09,995 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,995 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:09,995 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:09,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:09,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:09,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:09,995 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:09,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:09,997 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:09,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:09,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,000 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,000 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,000 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,000 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:10,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:10,000 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,000 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,000 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,000 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:10,000 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:10,000 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:10,000 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:10,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:10,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:10,000 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:10,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:10,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:10,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:10,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839030032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,100 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:10,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:10,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839030099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,101 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839030100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:10,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:10,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:10,104 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:10,104 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:10,104 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:10,105 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:10,105 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,105 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839030102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,107 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,107 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:10,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:10,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:10,108 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,108 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:10,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839030107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:10,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:10,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:10,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:10,109 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:10,109 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:10,110 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:10,110 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:10,110 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:10,110 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:10,110 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:10,110 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:10,111 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:10,111 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:10,111 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:10,111 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,111 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:10,111 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:10,112 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,113 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:10,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:10,113 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,113 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:10,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:10,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:10,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:10,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:10,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6684ac9f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:10,115 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:10,115 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,119 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,119 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:10,119 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:10,119 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:10,120 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,120 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:10,122 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,122 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:10,122 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:10,122 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:10,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:10,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:10,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:10,125 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:10,125 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:10,125 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:10,125 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:10,125 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:10,125 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:10,126 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,126 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:10,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,127 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:10,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:10,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:10,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,128 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:10,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:10,129 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,133 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:10,133 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:10,133 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:10,133 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:10,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:10,133 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,133 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,133 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,133 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,133 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,133 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:10,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:10,133 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:10,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:10,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:10,134 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:10,234 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:10,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:10,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:10,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:10,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:10,238 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:10,238 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:10,238 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:10,238 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:10,238 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,238 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,240 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:10,240 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:10,240 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,240 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:10,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:10,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:10,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:10,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:10,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:10,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:10,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:10,242 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:10,242 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:10,243 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:10,243 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:10,243 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:10,243 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:10,243 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:10,243 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,243 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:10,243 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:10,246 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,246 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:10,246 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,246 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:10,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:10,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:10,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@370d222d[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:10,248 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:10,248 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,249 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,250 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,250 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:10,250 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:10,250 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:10,250 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:10,251 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:10,251 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,251 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:10,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:10,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:10,251 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:10,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:10,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:10,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:10,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:10,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:10,254 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:10,254 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:10,254 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:10,254 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:10,254 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:10,256 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,256 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:10,256 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:10,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:10,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:10,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:10,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:10,258 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:10,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,260 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:10,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,263 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,263 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,263 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,263 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:10,263 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:10,264 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,263 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:10,264 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:10,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:10,264 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:10,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:10,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:10,264 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:10,264 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:10,264 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,322 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=19.73 KB at sequenceid=75 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/d222394ab046470f93ab07340fa2a8f7 2023-03-26 13:56:10,326 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/0993abfab80b410cbed406edc1a56926 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/0993abfab80b410cbed406edc1a56926 2023-03-26 13:56:10,331 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/0993abfab80b410cbed406edc1a56926, entries=250, sequenceid=75, filesize=38.7 K 2023-03-26 13:56:10,332 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/f128f6749e954bbab4f6564e311d4aad as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/f128f6749e954bbab4f6564e311d4aad 2023-03-26 13:56:10,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/f128f6749e954bbab4f6564e311d4aad, entries=150, sequenceid=75, filesize=11.6 K 2023-03-26 13:56:10,336 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/d222394ab046470f93ab07340fa2a8f7 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/d222394ab046470f93ab07340fa2a8f7 2023-03-26 13:56:10,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/d222394ab046470f93ab07340fa2a8f7, entries=150, sequenceid=75, filesize=11.6 K 2023-03-26 13:56:10,340 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~61.83 KB/63310, heapSize ~169.05 KB/173104, currentSize=120.76 KB/123660 for f904470a09c7c6a52bdab844746de048 in 909ms, sequenceid=75, compaction requested=true 2023-03-26 13:56:10,340 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:10,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:10,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:10,340 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:56:10,340 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:56:10,340 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:10,343 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 116868 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:56:10,343 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/A is initiating minor compaction (all files) 2023-03-26 13:56:10,343 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/A in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:10,343 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/90c2a16f4af84d9db1e0bb2bb45a144c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/8ac1b81b14e04108a8537bbc0811eb22, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/0ad62a672be04218ba8805695786ad7a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/0993abfab80b410cbed406edc1a56926] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=114.1 K 2023-03-26 13:56:10,344 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 90c2a16f4af84d9db1e0bb2bb45a144c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1679838968150 2023-03-26 13:56:10,344 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:56:10,344 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 8ac1b81b14e04108a8537bbc0811eb22, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1679838968166 2023-03-26 13:56:10,344 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/B is initiating minor compaction (all files) 2023-03-26 13:56:10,344 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/B in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:10,344 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1e7265ad98b34235a81b53dce03ed87d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1e2b333f69e64de48dd0c3180bdcc576, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/433907e55f86483582fa4b23cfc249ee, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/f128f6749e954bbab4f6564e311d4aad] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=46.4 K 2023-03-26 13:56:10,345 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 0ad62a672be04218ba8805695786ad7a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1679838968783 2023-03-26 13:56:10,345 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 1e7265ad98b34235a81b53dce03ed87d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1679838968150 2023-03-26 13:56:10,346 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 1e2b333f69e64de48dd0c3180bdcc576, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1679838968166 2023-03-26 13:56:10,346 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 0993abfab80b410cbed406edc1a56926, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1679838969418 2023-03-26 13:56:10,346 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 433907e55f86483582fa4b23cfc249ee, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1679838968783 2023-03-26 13:56:10,347 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting f128f6749e954bbab4f6564e311d4aad, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1679838969418 2023-03-26 13:56:10,358 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#B#compaction#595 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:10,364 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:10,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:10,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:10,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:10,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:10,372 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:10,372 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:10,372 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:10,372 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:10,373 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,373 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,374 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#A#compaction#594 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:10,374 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:10,374 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:10,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:10,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,375 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,375 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:10,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:10,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:10,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:10,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:10,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:10,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:10,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:10,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:10,376 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:10,377 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:10,377 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:10,377 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=120.76 KB heapSize=345.14 KB 2023-03-26 13:56:10,380 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:10,380 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:10,380 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:10,380 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:10,380 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:10,380 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:10,385 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/4af011ec655d44aa9360182413b3850e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/4af011ec655d44aa9360182413b3850e 2023-03-26 13:56:10,390 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f904470a09c7c6a52bdab844746de048/B of f904470a09c7c6a52bdab844746de048 into 4af011ec655d44aa9360182413b3850e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:10,390 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:10,390 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/B, priority=12, startTime=1679838970340; duration=0sec 2023-03-26 13:56:10,390 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:10,390 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:56:10,391 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:56:10,391 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/C is initiating minor compaction (all files) 2023-03-26 13:56:10,391 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/C in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:10,391 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/1d82424a4ddf466787044d6ba1570b6a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/02cddce57c2b4004bda9351afc72f0da, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/b20f49074f5049cfa424062eddd93c35, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/d222394ab046470f93ab07340fa2a8f7] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=46.4 K 2023-03-26 13:56:10,392 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 1d82424a4ddf466787044d6ba1570b6a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=17, earliestPutTs=1679838968150 2023-03-26 13:56:10,393 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 02cddce57c2b4004bda9351afc72f0da, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1679838968166 2023-03-26 13:56:10,394 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting b20f49074f5049cfa424062eddd93c35, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=49, earliestPutTs=1679838968783 2023-03-26 13:56:10,395 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting d222394ab046470f93ab07340fa2a8f7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1679838969418 2023-03-26 13:56:10,404 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#C#compaction#597 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:10,405 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303269255ecfef8364bd8be2cd1cb331b2ecd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303269255ecfef8364bd8be2cd1cb331b2ecd 2023-03-26 13:56:10,434 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=110, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/6d96e855229d4d69b99655ab956e85fd 2023-03-26 13:56:10,448 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/fbbd03c871144de4895e20978ebbf576 2023-03-26 13:56:10,476 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/c06cf98dd9754d5db6b5a999c88b335b 2023-03-26 13:56:10,481 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/6d96e855229d4d69b99655ab956e85fd as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/6d96e855229d4d69b99655ab956e85fd 2023-03-26 13:56:10,485 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/6d96e855229d4d69b99655ab956e85fd, entries=150, sequenceid=110, filesize=25.1 K 2023-03-26 13:56:10,485 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/fbbd03c871144de4895e20978ebbf576 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/fbbd03c871144de4895e20978ebbf576 2023-03-26 13:56:10,489 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/fbbd03c871144de4895e20978ebbf576, entries=150, sequenceid=110, filesize=11.6 K 2023-03-26 13:56:10,489 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/c06cf98dd9754d5db6b5a999c88b335b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/c06cf98dd9754d5db6b5a999c88b335b 2023-03-26 13:56:10,493 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/c06cf98dd9754d5db6b5a999c88b335b, entries=150, sequenceid=110, filesize=11.6 K 2023-03-26 13:56:10,493 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~345.09 KB/353376, currentSize=0 B/0 for f904470a09c7c6a52bdab844746de048 in 116ms, sequenceid=110, compaction requested=false 2023-03-26 13:56:10,493 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:10,493 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:10,493 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:10,493 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:10,493 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,494 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:10,494 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:10,495 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,495 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:10,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:10,495 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,495 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:10,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:10,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:10,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:10,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:10,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@410091d5[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:10,497 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:10,497 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,498 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,498 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:10,498 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:10,498 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:10,498 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,498 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:10,501 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:10,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:10,501 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:10,501 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:10,501 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:10,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:10,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:10,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:10,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:10,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:10,504 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:10,504 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:10,504 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:10,504 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:10,504 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:10,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:10,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:10,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:10,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:10,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:10,506 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:10,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:10,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,507 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:10,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,509 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:10,509 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:10,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:10,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:10,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:10,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:10,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:10,510 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:10,509 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:10,510 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:10,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:10,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:10,509 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,510 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,510 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,510 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,510 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,510 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,510 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,583 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-26 13:56:10,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:10,599 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:56:10,600 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:10,600 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:10,600 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:10,600 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:10,600 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:10,600 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:10,610 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:10,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:10,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:10,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:10,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:10,620 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:10,620 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:10,620 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:10,620 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:10,620 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,620 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,622 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:10,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:10,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:10,622 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,622 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,622 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:10,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:10,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:10,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:10,628 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:10,628 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:10,628 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:10,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838970621, max timestamp=1679838970630Num uniques 859; 2023-03-26 13:56:10,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838970621, max timestamp=1679838970630Num uniques 859; 2023-03-26 13:56:10,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838970621, max timestamp=1679838970630Num uniques 859; 2023-03-26 13:56:10,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:10,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:10,632 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:10,632 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:10,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:10,634 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:10,634 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:10,638 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:10,638 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:10,639 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:10,639 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:10,639 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,649 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:10,650 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:10,652 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,652 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:10,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:10,653 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,653 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:10,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:10,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:10,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3de1e0be[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:10,654 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:10,654 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,656 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,656 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:10,656 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:10,656 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:10,656 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,657 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:10,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839030650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839030651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,660 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:10,660 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:10,660 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:10,660 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:10,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:10,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839030655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,660 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326ab23e4033d264f3f91eaba2b077f22fc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ab23e4033d264f3f91eaba2b077f22fc 2023-03-26 13:56:10,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:10,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:10,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839030654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:10,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:10,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:10,663 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:10,663 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:10,663 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:10,663 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:10,663 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:10,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839030657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,666 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,666 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:10,666 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:10,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:10,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,666 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:10,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:10,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:10,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:10,667 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:10,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,670 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:10,670 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:10,670 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:10,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:10,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:10,670 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,671 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,671 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,671 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,671 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:10,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:10,671 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,671 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:10,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:10,671 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:10,671 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:10,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:10,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839030758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839030759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,763 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839030762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,767 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839030763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839030765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,771 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:10,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:10,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:10,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:10,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:10,775 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:10,775 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:10,775 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:10,775 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:10,775 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,775 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,777 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,777 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:10,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:10,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:10,777 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,777 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:10,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:10,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:10,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:10,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:10,784 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:10,784 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:10,784 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:10,792 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:10,792 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:10,792 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:10,792 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:10,792 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:10,792 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:10,792 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:10,792 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:10,792 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,792 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:10,793 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:10,794 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:10,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:10,794 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:10,795 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,795 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:10,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:10,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:10,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:10,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e4243c9[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:10,796 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:10,796 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,799 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,799 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:10,799 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:10,799 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:10,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:10,801 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,801 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:10,801 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:10,801 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:10,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:10,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:10,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:10,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:10,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:10,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:10,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:10,805 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:10,805 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:10,805 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:10,805 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:10,805 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:10,807 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,807 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:10,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:10,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:10,808 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:10,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:10,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:10,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,809 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:10,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,812 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:10,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:10,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,814 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:10,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:10,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:10,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:10,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:10,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:10,815 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:10,815 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:10,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:10,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:10,837 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/5062db6f30414173bd1d3822a1c577fe as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5062db6f30414173bd1d3822a1c577fe 2023-03-26 13:56:10,843 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/1e424e85b5a44957adf6c8327ec5c781 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/1e424e85b5a44957adf6c8327ec5c781 2023-03-26 13:56:10,845 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f904470a09c7c6a52bdab844746de048/C of f904470a09c7c6a52bdab844746de048 into 5062db6f30414173bd1d3822a1c577fe(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:10,845 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:10,846 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/C, priority=12, startTime=1679838970340; duration=0sec 2023-03-26 13:56:10,846 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:10,849 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f904470a09c7c6a52bdab844746de048/A of f904470a09c7c6a52bdab844746de048 into 1e424e85b5a44957adf6c8327ec5c781(size=25.3 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:10,849 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:10,849 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/A, priority=12, startTime=1679838970340; duration=0sec 2023-03-26 13:56:10,849 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:10,915 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:10,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:10,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:10,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:10,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:10,924 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:10,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:10,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:10,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:10,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,927 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:10,927 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:10,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:10,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:10,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:10,928 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,928 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:10,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:10,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:10,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:10,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:10,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:10,932 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:10,932 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:10,932 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:10,932 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:10,933 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:10,933 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:10,933 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:10,933 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:10,933 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,933 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:10,933 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:10,934 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:10,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:10,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:10,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:10,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:10,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20800a66[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:10,937 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:10,937 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,938 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,938 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:10,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,939 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:10,939 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:10,939 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:10,941 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:10,941 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:10,941 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:10,941 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:10,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:10,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:10,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:10,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:10,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:10,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:10,944 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:10,944 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:10,944 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:10,944 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:10,944 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:10,947 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,947 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:10,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,947 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:10,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:10,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:10,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:10,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,949 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:10,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,950 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:10,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:10,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,953 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:10,953 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:10,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:10,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:10,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:10,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:10,954 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,954 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,954 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:10,954 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,954 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,954 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:10,954 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:10,954 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:10,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:10,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:10,955 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:10,955 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:10,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:10,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839030963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839030963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839030966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839030968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:10,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:10,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839030969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,055 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:11,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:11,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:11,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:11,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:11,058 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:11,058 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:11,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:11,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:11,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,061 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:11,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:11,061 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:11,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,061 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,061 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:11,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:11,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,062 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:11,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:11,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:11,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:11,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:11,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:11,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:11,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:11,062 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:11,063 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:11,063 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:11,063 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:11,063 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:11,063 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:11,063 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,063 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:11,063 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:11,065 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,065 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:11,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:11,065 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,065 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:11,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:11,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:11,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:11,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:11,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65cbb99a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:11,067 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:11,067 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,070 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,070 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:11,070 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:11,070 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:11,070 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,070 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:11,072 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:11,072 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,072 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:11,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:11,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:11,072 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:11,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:11,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:11,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:11,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:11,075 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:11,075 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:11,075 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:11,075 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:11,075 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:11,077 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,077 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:11,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:11,077 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:11,078 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:11,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:11,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,080 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:11,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,083 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=121, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/a4d8c6e1c4174296bc54e44b9d0ace30 2023-03-26 13:56:11,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:11,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:11,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:11,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:11,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,086 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,087 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,087 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,087 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,087 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,087 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:11,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:11,087 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:11,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:11,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:11,087 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:11,087 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:11,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:11,187 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:11,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:11,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:11,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:11,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:11,191 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:11,191 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:11,191 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:11,191 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:11,192 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,192 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,194 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:11,194 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:11,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:11,194 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,194 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:11,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,195 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:11,195 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:11,195 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:11,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:11,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:11,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:11,195 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:11,196 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:11,196 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:11,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:11,196 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:11,196 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:11,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:11,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:11,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:11,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:11,198 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:11,198 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:11,198 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,198 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:11,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:11,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@579d518d[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:11,200 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:11,200 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,201 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,201 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,202 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:11,201 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:11,202 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:11,202 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:11,204 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:11,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:11,204 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:11,204 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:11,204 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:11,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:11,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:11,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:11,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:11,207 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:11,207 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:11,207 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:11,207 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:11,207 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:11,209 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:11,209 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,209 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:11,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:11,210 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:11,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,211 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:11,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:11,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:11,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:11,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:11,214 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:11,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:11,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,215 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:11,215 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:11,215 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,215 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,215 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,215 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,215 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:11,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:11,269 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:11,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839031266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:11,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839031268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:11,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839031270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:11,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839031271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:11,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839031273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,315 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:11,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:11,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:11,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:11,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:11,319 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:11,319 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:11,319 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:11,319 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:11,319 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,319 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,320 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:11,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:11,321 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,322 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,322 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:11,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:11,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:11,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:11,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:11,323 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:11,323 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:11,323 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:11,323 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:11,323 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:11,323 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:11,323 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:11,324 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:11,324 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:11,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:11,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:11,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:11,324 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:11,326 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:11,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:11,326 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:11,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,327 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:11,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b2d0cbc[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:11,328 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:11,328 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,329 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,329 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:11,330 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:11,330 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:11,331 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:11,331 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:11,331 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,331 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:11,331 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:11,331 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:11,331 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:11,331 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:11,331 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:11,331 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:11,331 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:11,334 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,334 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,334 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:11,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:11,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:11,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,335 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,336 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,340 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,340 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,340 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,340 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,341 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:11,341 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:11,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:11,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:11,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:11,341 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:11,341 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:11,341 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:11,341 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,341 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,341 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:11,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:11,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:11,441 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:11,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:11,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:11,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:11,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:11,445 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:11,445 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:11,445 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:11,445 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:11,446 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,446 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,448 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:11,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:11,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:11,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,448 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:11,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:11,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:11,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:11,449 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:11,449 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:11,449 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,449 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:11,449 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:11,449 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:11,449 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:11,449 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:11,450 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:11,450 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:11,450 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:11,450 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:11,450 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:11,450 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,450 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:11,450 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:11,451 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:11,452 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:11,452 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,452 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:11,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:11,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:11,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bdef285[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:11,454 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:11,454 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,455 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,455 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,455 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:11,455 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:11,455 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:11,455 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:11,457 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:11,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,459 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:11,459 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:11,459 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:11,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:11,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:11,460 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:11,460 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:11,460 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:11,460 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:11,460 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:11,460 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:11,461 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,461 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:11,461 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:11,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:11,463 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:11,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,464 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:11,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,469 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:11,469 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,469 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,469 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,469 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,469 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,469 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:11,469 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,469 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:11,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:11,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:11,470 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:11,469 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:11,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:11,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:11,470 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:11,470 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:11,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:11,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:11,499 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/84a002c15dba4981ac1d65b94554a8bb 2023-03-26 13:56:11,518 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=121 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/df30b84ccabf43c0bc147c20ecd25794 2023-03-26 13:56:11,523 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/a4d8c6e1c4174296bc54e44b9d0ace30 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/a4d8c6e1c4174296bc54e44b9d0ace30 2023-03-26 13:56:11,527 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/a4d8c6e1c4174296bc54e44b9d0ace30, entries=250, sequenceid=121, filesize=38.7 K 2023-03-26 13:56:11,528 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/84a002c15dba4981ac1d65b94554a8bb as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/84a002c15dba4981ac1d65b94554a8bb 2023-03-26 13:56:11,532 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/84a002c15dba4981ac1d65b94554a8bb, entries=150, sequenceid=121, filesize=11.6 K 2023-03-26 13:56:11,532 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/df30b84ccabf43c0bc147c20ecd25794 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/df30b84ccabf43c0bc147c20ecd25794 2023-03-26 13:56:11,537 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/df30b84ccabf43c0bc147c20ecd25794, entries=150, sequenceid=121, filesize=11.6 K 2023-03-26 13:56:11,538 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f904470a09c7c6a52bdab844746de048 in 939ms, sequenceid=121, compaction requested=true 2023-03-26 13:56:11,538 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:11,538 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:11,538 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:56:11,538 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:56:11,538 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:11,538 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:11,539 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91260 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:56:11,539 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/A is initiating minor compaction (all files) 2023-03-26 13:56:11,540 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/A in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:11,540 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/1e424e85b5a44957adf6c8327ec5c781, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/6d96e855229d4d69b99655ab956e85fd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/a4d8c6e1c4174296bc54e44b9d0ace30] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=89.1 K 2023-03-26 13:56:11,540 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:56:11,540 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/B is initiating minor compaction (all files) 2023-03-26 13:56:11,540 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/B in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:11,540 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/4af011ec655d44aa9360182413b3850e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/fbbd03c871144de4895e20978ebbf576, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/84a002c15dba4981ac1d65b94554a8bb] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=35.0 K 2023-03-26 13:56:11,540 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 1e424e85b5a44957adf6c8327ec5c781, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1679838969418 2023-03-26 13:56:11,541 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 4af011ec655d44aa9360182413b3850e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1679838969418 2023-03-26 13:56:11,541 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 6d96e855229d4d69b99655ab956e85fd, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679838969480 2023-03-26 13:56:11,541 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting fbbd03c871144de4895e20978ebbf576, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679838969480 2023-03-26 13:56:11,541 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting a4d8c6e1c4174296bc54e44b9d0ace30, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679838970575 2023-03-26 13:56:11,541 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 84a002c15dba4981ac1d65b94554a8bb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679838970582 2023-03-26 13:56:11,549 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#B#compaction#604 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:11,557 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#A#compaction#603 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:11,570 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:11,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:11,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:11,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:11,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:11,576 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:11,576 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:11,576 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:11,576 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:11,576 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,577 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,579 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:11,579 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:11,579 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,579 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:11,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:11,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:11,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:11,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:11,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:11,604 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:11,604 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:11,604 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:11,604 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:11,604 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:11,605 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:11,605 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:11,605 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-26 13:56:11,605 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:11,605 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:11,605 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:11,605 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:11,605 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:11,605 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:11,609 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/53eb9150507c4541953a4cfd1043eda0 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/53eb9150507c4541953a4cfd1043eda0 2023-03-26 13:56:11,614 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f904470a09c7c6a52bdab844746de048/B of f904470a09c7c6a52bdab844746de048 into 53eb9150507c4541953a4cfd1043eda0(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:11,614 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:11,615 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/B, priority=13, startTime=1679838971538; duration=0sec 2023-03-26 13:56:11,615 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:11,615 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:56:11,618 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:56:11,618 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/C is initiating minor compaction (all files) 2023-03-26 13:56:11,618 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/C in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:11,618 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5062db6f30414173bd1d3822a1c577fe, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/c06cf98dd9754d5db6b5a999c88b335b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/df30b84ccabf43c0bc147c20ecd25794] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=35.0 K 2023-03-26 13:56:11,618 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 5062db6f30414173bd1d3822a1c577fe, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=75, earliestPutTs=1679838969418 2023-03-26 13:56:11,618 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting c06cf98dd9754d5db6b5a999c88b335b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679838969480 2023-03-26 13:56:11,619 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting df30b84ccabf43c0bc147c20ecd25794, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679838970582 2023-03-26 13:56:11,624 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#C#compaction#606 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:11,632 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303269d17e50fde8d40ff9e23741b54f35dbd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303269d17e50fde8d40ff9e23741b54f35dbd 2023-03-26 13:56:11,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:11,773 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:11,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:11,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839031778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:11,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839031778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:11,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839031781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:11,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839031782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:11,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839031782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:11,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839031883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:11,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839031884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:11,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839031887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:11,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:11,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839031887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,039 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/5cd34c17d1124e03892cf184b7ecfb5b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5cd34c17d1124e03892cf184b7ecfb5b 2023-03-26 13:56:12,045 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f904470a09c7c6a52bdab844746de048/C of f904470a09c7c6a52bdab844746de048 into 5cd34c17d1124e03892cf184b7ecfb5b(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:12,045 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:12,045 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/C, priority=13, startTime=1679838971538; duration=0sec 2023-03-26 13:56:12,045 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:12,048 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=160, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/42fc1c46c1814651bc29473e0a7273db 2023-03-26 13:56:12,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:12,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839032086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,088 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:12,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839032086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:12,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839032090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:12,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839032090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:12,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839032389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,394 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:12,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839032392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:12,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839032393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,395 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:12,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839032393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,412 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/6aee5876e3a246ed9628a2d3e8b747fd as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/6aee5876e3a246ed9628a2d3e8b747fd 2023-03-26 13:56:12,416 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f904470a09c7c6a52bdab844746de048/A of f904470a09c7c6a52bdab844746de048 into 6aee5876e3a246ed9628a2d3e8b747fd(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:12,417 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:12,417 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/A, priority=13, startTime=1679838971538; duration=0sec 2023-03-26 13:56:12,417 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:12,462 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/45c74fbd7d9540cfa34b36dc866f99b1 2023-03-26 13:56:12,478 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/66e4352320ce4b68b8383a950f941729 2023-03-26 13:56:12,484 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/42fc1c46c1814651bc29473e0a7273db as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/42fc1c46c1814651bc29473e0a7273db 2023-03-26 13:56:12,488 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/42fc1c46c1814651bc29473e0a7273db, entries=150, sequenceid=160, filesize=25.3 K 2023-03-26 13:56:12,489 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/45c74fbd7d9540cfa34b36dc866f99b1 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/45c74fbd7d9540cfa34b36dc866f99b1 2023-03-26 13:56:12,493 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/45c74fbd7d9540cfa34b36dc866f99b1, entries=150, sequenceid=160, filesize=11.8 K 2023-03-26 13:56:12,493 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/66e4352320ce4b68b8383a950f941729 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/66e4352320ce4b68b8383a950f941729 2023-03-26 13:56:12,501 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/66e4352320ce4b68b8383a950f941729, entries=150, sequenceid=160, filesize=11.8 K 2023-03-26 13:56:12,502 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=53.67 KB/54960 for f904470a09c7c6a52bdab844746de048 in 897ms, sequenceid=160, compaction requested=false 2023-03-26 13:56:12,502 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:12,502 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:12,502 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:12,502 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:12,502 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,502 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:12,502 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:12,504 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:12,504 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:12,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:12,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:12,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:12,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23618f0d[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:12,507 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:12,507 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:12,508 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:12,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:12,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:12,508 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:12,508 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:12,508 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:12,508 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,508 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:12,510 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:12,510 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:12,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:12,510 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:12,511 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:12,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:12,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:12,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:12,513 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:12,513 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:12,513 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:12,513 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:12,513 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:12,516 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:12,516 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:12,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:12,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:12,516 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:12,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:12,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:12,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:12,516 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:12,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:12,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:12,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:12,519 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:12,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:12,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,521 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:12,521 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,521 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:12,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:12,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:12,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:12,521 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:12,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:12,522 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:12,521 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:12,522 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:12,522 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:12,522 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,522 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:12,522 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:12,522 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:12,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:12,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:12,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:12,622 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:12,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:12,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:12,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:12,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:12,625 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:12,625 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:12,625 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:12,625 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:12,625 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:12,626 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:12,627 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:12,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:12,627 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,627 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,627 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:12,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:12,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:12,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:12,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:12,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:12,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:12,629 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:12,629 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:12,629 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:12,629 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:12,629 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:12,629 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:12,629 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:12,629 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 13:56:12,630 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:12,630 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:12,630 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:12,630 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:12,630 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:12,630 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:12,653 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032612ddbf404d6447fda82f6cae81ed94b1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032612ddbf404d6447fda82f6cae81ed94b1 2023-03-26 13:56:12,658 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=174, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/439eafdc7d6a4360bb0a1a371681af6a 2023-03-26 13:56:12,668 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/d8eb7e553aef4cc89784022a8e1c3e3c 2023-03-26 13:56:12,679 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/6e2f8f86530b44b292e5a106604e4c88 2023-03-26 13:56:12,684 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/439eafdc7d6a4360bb0a1a371681af6a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/439eafdc7d6a4360bb0a1a371681af6a 2023-03-26 13:56:12,687 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/439eafdc7d6a4360bb0a1a371681af6a, entries=100, sequenceid=174, filesize=18.5 K 2023-03-26 13:56:12,688 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/d8eb7e553aef4cc89784022a8e1c3e3c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/d8eb7e553aef4cc89784022a8e1c3e3c 2023-03-26 13:56:12,691 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/d8eb7e553aef4cc89784022a8e1c3e3c, entries=100, sequenceid=174, filesize=9.4 K 2023-03-26 13:56:12,691 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/6e2f8f86530b44b292e5a106604e4c88 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/6e2f8f86530b44b292e5a106604e4c88 2023-03-26 13:56:12,695 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/6e2f8f86530b44b292e5a106604e4c88, entries=100, sequenceid=174, filesize=9.4 K 2023-03-26 13:56:12,696 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for f904470a09c7c6a52bdab844746de048 in 66ms, sequenceid=174, compaction requested=true 2023-03-26 13:56:12,696 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:12,696 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:12,696 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:12,696 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:12,696 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,696 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:12,696 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:12,697 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,697 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:12,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:12,698 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:12,698 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:12,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:12,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:12,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ecda3d5[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:12,700 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:12,700 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:12,702 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:12,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:12,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:12,702 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:12,702 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:12,702 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:12,702 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,702 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:12,704 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:12,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:12,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:12,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:12,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:12,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:12,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:12,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:12,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:12,707 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:12,707 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:12,707 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:12,707 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:12,707 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:12,708 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:12,708 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:12,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:12,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:12,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:12,709 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:12,710 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,712 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:12,712 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:12,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:12,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:12,712 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:12,712 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:12,712 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:12,712 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,712 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:12,712 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:12,712 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:12,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:12,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:12,712 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:12,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:12,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:12,713 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:12,713 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:12,712 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:12,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:12,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:12,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:12,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:12,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:12,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:12,813 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:12,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:12,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:12,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:12,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:12,816 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:12,816 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:12,816 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:12,816 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:12,817 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:12,817 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:12,819 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:12,819 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:12,819 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,819 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:12,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:12,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:12,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:12,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:12,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:12,821 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:12,821 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:12,821 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:12,821 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:12,821 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:12,821 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:12,821 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:12,821 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-26 13:56:12,822 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:12,822 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:12,822 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:12,822 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:12,822 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:12,822 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:12,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:12,828 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:12,856 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303260cfdebd83f714483b89c448664676823 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260cfdebd83f714483b89c448664676823 2023-03-26 13:56:12,862 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=182, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/e2c67069d15e4d4ebfff483b04823381 2023-03-26 13:56:12,877 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/91ad5684e0bb4e188f1e34b1b92d3a7c 2023-03-26 13:56:12,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838972859, max timestamp=1679838972877Num uniques 859; 2023-03-26 13:56:12,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:12,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838972859, max timestamp=1679838972877Num uniques 859; 2023-03-26 13:56:12,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838972859, max timestamp=1679838972877Num uniques 859; 2023-03-26 13:56:12,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:12,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:12,912 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1679838972901, max timestamp=1679838972910Num uniques 1018; 2023-03-26 13:56:12,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=16848, suffix heap size=133840,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=368 2023-03-26 13:56:12,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1679838972901, max timestamp=1679838972910Num uniques 1018; 2023-03-26 13:56:12,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1679838972901, max timestamp=1679838972910Num uniques 1018; 2023-03-26 13:56:12,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=16848, suffix heap size=133840,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=368 2023-03-26 13:56:12,913 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=16848, suffix heap size=133840,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=368 2023-03-26 13:56:12,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:12,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839032924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:12,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:12,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839032926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839032926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:12,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839032927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:12,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:12,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839032928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839033029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839033032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839033032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839033032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839033032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,234 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839033232, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839033235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839033235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839033235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839033236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,290 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=182 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/504a9b4a505641ad933963bae766d958 2023-03-26 13:56:13,293 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/e2c67069d15e4d4ebfff483b04823381 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/e2c67069d15e4d4ebfff483b04823381 2023-03-26 13:56:13,298 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/e2c67069d15e4d4ebfff483b04823381, entries=200, sequenceid=182, filesize=32.1 K 2023-03-26 13:56:13,299 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/91ad5684e0bb4e188f1e34b1b92d3a7c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/91ad5684e0bb4e188f1e34b1b92d3a7c 2023-03-26 13:56:13,301 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/91ad5684e0bb4e188f1e34b1b92d3a7c, entries=150, sequenceid=182, filesize=11.8 K 2023-03-26 13:56:13,302 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/504a9b4a505641ad933963bae766d958 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/504a9b4a505641ad933963bae766d958 2023-03-26 13:56:13,305 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/504a9b4a505641ad933963bae766d958, entries=150, sequenceid=182, filesize=11.8 K 2023-03-26 13:56:13,306 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=161.02 KB/164880 for f904470a09c7c6a52bdab844746de048 in 485ms, sequenceid=182, compaction requested=true 2023-03-26 13:56:13,306 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:13,306 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:13,306 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:13,306 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:13,306 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,306 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:13,306 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:13,308 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,308 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:13,308 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:13,308 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:13,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:13,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:13,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:13,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:13,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d56145c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:13,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:13,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:13,311 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:13,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:13,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:13,311 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:13,311 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:13,311 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:13,311 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,311 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:13,312 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:13,312 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:13,312 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,313 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:13,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:13,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:13,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:13,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:13,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:13,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:13,315 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:13,315 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:13,315 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:13,315 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:13,315 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:13,318 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:13,318 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:13,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:13,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:13,318 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:13,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:13,319 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:13,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:13,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,320 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:13,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:13,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,322 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:13,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:13,322 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:13,322 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:13,322 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:13,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:13,322 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:13,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:13,322 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:13,322 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:13,323 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:13,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:13,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:13,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:13,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:13,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:13,423 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:13,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:13,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:13,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:13,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:13,426 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:13,426 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:13,426 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:13,426 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:13,427 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:13,427 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:13,431 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,431 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:13,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:13,431 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,431 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:13,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:13,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:13,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:13,432 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:13,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:13,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:13,432 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:13,433 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:13,433 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:13,433 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:13,433 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:13,433 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:13,433 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:13,433 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=161.02 KB heapSize=445.08 KB 2023-03-26 13:56:13,433 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:13,433 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:13,433 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:13,433 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:13,434 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:13,434 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:13,453 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303260167229dbb524db69b6dcaeead4ca5f3 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260167229dbb524db69b6dcaeead4ca5f3 2023-03-26 13:56:13,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:13,537 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:13,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839033540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839033541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839033541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839033543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,546 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839033543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,647 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839033644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839033647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839033647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839033848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839033849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:13,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839033850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:13,858 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=236, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/68eb1c0dade24dbb9eeed2f9de630e15 2023-03-26 13:56:14,048 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:14,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46364 deadline: 1679839034047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:14,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839034048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:14,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:14,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839034151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839034151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:14,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839034154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,268 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/54f4bfff636845a19f1efc1dfbff70b8 2023-03-26 13:56:14,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:14,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839034657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:14,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839034659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:14,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839034659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,677 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/7e16d5539a624e918b35c44409d7112a 2023-03-26 13:56:14,681 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/68eb1c0dade24dbb9eeed2f9de630e15 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/68eb1c0dade24dbb9eeed2f9de630e15 2023-03-26 13:56:14,684 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/68eb1c0dade24dbb9eeed2f9de630e15, entries=150, sequenceid=236, filesize=25.3 K 2023-03-26 13:56:14,684 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/54f4bfff636845a19f1efc1dfbff70b8 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/54f4bfff636845a19f1efc1dfbff70b8 2023-03-26 13:56:14,688 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/54f4bfff636845a19f1efc1dfbff70b8, entries=150, sequenceid=236, filesize=11.8 K 2023-03-26 13:56:14,688 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/7e16d5539a624e918b35c44409d7112a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/7e16d5539a624e918b35c44409d7112a 2023-03-26 13:56:14,691 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/7e16d5539a624e918b35c44409d7112a, entries=150, sequenceid=236, filesize=11.8 K 2023-03-26 13:56:14,692 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~445.03 KB/455712, currentSize=33.54 KB/34350 for f904470a09c7c6a52bdab844746de048 in 1259ms, sequenceid=236, compaction requested=true 2023-03-26 13:56:14,692 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:14,692 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:14,692 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:14,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:14,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:14,693 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:14,695 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:14,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:14,696 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:14,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:14,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21708da9[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:14,697 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:14,697 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:14,698 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:14,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:14,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:14,698 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:14,698 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:14,698 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:14,698 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,698 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:14,700 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:14,700 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:14,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:14,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:14,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:14,703 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:14,703 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:14,703 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:14,703 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:14,703 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:14,705 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:14,705 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:14,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:14,705 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:14,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:14,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:14,705 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:14,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:14,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:14,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:14,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,707 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:14,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:14,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,710 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,710 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:14,710 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:14,710 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:14,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:14,710 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:14,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:14,710 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,710 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:14,710 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:14,710 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:14,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:14,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:14,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:14,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:14,710 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:14,711 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:14,711 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:14,811 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:14,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:14,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:14,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:14,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:14,814 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:14,814 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:14,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:14,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:14,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:14,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:14,816 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:14,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:14,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:14,816 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,817 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:14,817 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:14,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:14,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:14,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:14,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:14,818 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:14,818 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:14,818 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:14,818 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:14,818 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:14,818 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:14,818 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:14,818 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-26 13:56:14,818 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:14,818 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:14,818 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:14,819 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:14,819 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:14,819 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:14,833 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326a6565602e610409a910620c92ee321b2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a6565602e610409a910620c92ee321b2 2023-03-26 13:56:15,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:15,059 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:15,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838975081, max timestamp=1679838975088Num uniques 859; 2023-03-26 13:56:15,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:15,090 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838975081, max timestamp=1679838975088Num uniques 859; 2023-03-26 13:56:15,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:15,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838975081, max timestamp=1679838975088Num uniques 859; 2023-03-26 13:56:15,091 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:15,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:15,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679838975141, max timestamp=1679838975153Num uniques 1068; 2023-03-26 13:56:15,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-26 13:56:15,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679838975141, max timestamp=1679838975153Num uniques 1068; 2023-03-26 13:56:15,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839035151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-26 13:56:15,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679838975141, max timestamp=1679838975153Num uniques 1068; 2023-03-26 13:56:15,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-26 13:56:15,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679838975201, max timestamp=1679838975217Num uniques 1027; 2023-03-26 13:56:15,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-26 13:56:15,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679838975201, max timestamp=1679838975217Num uniques 1027; 2023-03-26 13:56:15,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679838975201, max timestamp=1679838975217Num uniques 1027; 2023-03-26 13:56:15,219 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-26 13:56:15,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-26 13:56:15,243 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=244, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/faa103eddba3441e8331cfd71d1851d7 2023-03-26 13:56:15,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679838975252, max timestamp=1679838975267Num uniques 1036; 2023-03-26 13:56:15,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-26 13:56:15,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679838975252, max timestamp=1679838975267Num uniques 1036; 2023-03-26 13:56:15,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-26 13:56:15,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679838975252, max timestamp=1679838975267Num uniques 1036; 2023-03-26 13:56:15,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-03-26 13:56:15,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:15,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46380 deadline: 1679839035297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679838975289, max timestamp=1679838975298Num uniques 1095; 2023-03-26 13:56:15,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679838975289, max timestamp=1679838975298Num uniques 1095; 2023-03-26 13:56:15,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679838975289, max timestamp=1679838975298Num uniques 1095; 2023-03-26 13:56:15,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-26 13:56:15,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-26 13:56:15,301 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-26 13:56:15,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679838975367, max timestamp=1679838975379Num uniques 1054; 2023-03-26 13:56:15,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-26 13:56:15,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679838975367, max timestamp=1679838975379Num uniques 1104; 2023-03-26 13:56:15,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679838975367, max timestamp=1679838975379Num uniques 1054; 2023-03-26 13:56:15,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-26 13:56:15,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-26 13:56:15,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679838975410, max timestamp=1679838975428Num uniques 1013; 2023-03-26 13:56:15,430 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-26 13:56:15,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679838975410, max timestamp=1679838975428Num uniques 1013; 2023-03-26 13:56:15,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679838975410, max timestamp=1679838975428Num uniques 1013; 2023-03-26 13:56:15,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-26 13:56:15,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-26 13:56:15,472 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679838975452, max timestamp=1679838975470Num uniques 1022; 2023-03-26 13:56:15,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-03-26 13:56:15,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679838975452, max timestamp=1679838975470Num uniques 1022; 2023-03-26 13:56:15,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679838975452, max timestamp=1679838975470Num uniques 1022; 2023-03-26 13:56:15,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-03-26 13:56:15,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10162, suffix heap size=134256,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=222 2023-03-26 13:56:15,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679838975499, max timestamp=1679838975511Num uniques 1081; 2023-03-26 13:56:15,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679838975499, max timestamp=1679838975511Num uniques 1081; 2023-03-26 13:56:15,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-03-26 13:56:15,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679838975499, max timestamp=1679838975511Num uniques 1081; 2023-03-26 13:56:15,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-03-26 13:56:15,522 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-03-26 13:56:15,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679838975547, max timestamp=1679838975555Num uniques 1090; 2023-03-26 13:56:15,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-26 13:56:15,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679838975547, max timestamp=1679838975555Num uniques 1090; 2023-03-26 13:56:15,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-26 13:56:15,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679838975547, max timestamp=1679838975555Num uniques 1090; 2023-03-26 13:56:15,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-26 13:56:15,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679838975570, max timestamp=1679838975589Num uniques 1049; 2023-03-26 13:56:15,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-03-26 13:56:15,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679838975570, max timestamp=1679838975589Num uniques 1049; 2023-03-26 13:56:15,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-03-26 13:56:15,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679838975570, max timestamp=1679838975589Num uniques 1049; 2023-03-26 13:56:15,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-03-26 13:56:15,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:15,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839035626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,631 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:15,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839035627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,651 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/661c919c39bd4b048be6bd60de91b8dc 2023-03-26 13:56:15,660 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=244 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/592b3b19d7eb4e1090097d80a0c03c77 2023-03-26 13:56:15,663 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/faa103eddba3441e8331cfd71d1851d7 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/faa103eddba3441e8331cfd71d1851d7 2023-03-26 13:56:15,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:15,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839035662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:15,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839035663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,666 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/faa103eddba3441e8331cfd71d1851d7, entries=100, sequenceid=244, filesize=18.5 K 2023-03-26 13:56:15,667 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:15,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839035664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,667 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/661c919c39bd4b048be6bd60de91b8dc as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/661c919c39bd4b048be6bd60de91b8dc 2023-03-26 13:56:15,670 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/661c919c39bd4b048be6bd60de91b8dc, entries=100, sequenceid=244, filesize=9.4 K 2023-03-26 13:56:15,671 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/592b3b19d7eb4e1090097d80a0c03c77 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/592b3b19d7eb4e1090097d80a0c03c77 2023-03-26 13:56:15,674 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/592b3b19d7eb4e1090097d80a0c03c77, entries=100, sequenceid=244, filesize=9.4 K 2023-03-26 13:56:15,675 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for f904470a09c7c6a52bdab844746de048 in 857ms, sequenceid=244, compaction requested=true 2023-03-26 13:56:15,675 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:15,675 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:15,675 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:15,675 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:15,675 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,675 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:15,675 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:15,677 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,677 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:15,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:15,678 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,678 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:15,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:15,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:15,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:15,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:15,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e46776d[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:15,679 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:15,679 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,680 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:15,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:15,680 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,681 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:15,680 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:15,682 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:15,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:15,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:15,682 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:15,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:15,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:15,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:15,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:15,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:15,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:15,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:15,684 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:15,684 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:15,684 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:15,684 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:15,684 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:15,686 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,686 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:15,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,686 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:15,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:15,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:15,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:15,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,688 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:15,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:15,689 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:15,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,693 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,693 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:15,693 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:15,693 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:15,693 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:15,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:15,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:15,693 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,693 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:15,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:15,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:15,693 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,693 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:15,693 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,693 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,693 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:15,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:15,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:15,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:15,735 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=154.31 KB heapSize=437.02 KB 2023-03-26 13:56:15,735 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:15,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-26 13:56:15,735 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:15,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-26 13:56:15,735 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:15,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-26 13:56:15,747 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303265ce1a662100b42db9029d9e3994b4d38 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303265ce1a662100b42db9029d9e3994b4d38 2023-03-26 13:56:15,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:15,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839035743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,748 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:15,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839035745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,753 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=454, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/e1aea2a8e93c46c8a2e7394f8e52de01 2023-03-26 13:56:15,793 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:15,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:15,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:15,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:15,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:15,798 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:15,798 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:15,798 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:15,798 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:15,798 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,798 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:15,800 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:15,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:15,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:15,801 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,801 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:15,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:15,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:15,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:15,802 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:15,802 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:15,802 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:15,802 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:15,802 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:15,802 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:15,802 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:15,802 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:15,802 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:15,802 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:15,802 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:15,802 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,802 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:15,802 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:15,804 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:15,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:15,804 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:15,805 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,805 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:15,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:15,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:15,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d66ae7b[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:15,806 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:15,806 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,808 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,808 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:15,808 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:15,808 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:15,808 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,809 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:15,811 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,811 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:15,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:15,811 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:15,811 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:15,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:15,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:15,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:15,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:15,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:15,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:15,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:15,814 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:15,814 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:15,814 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:15,814 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:15,814 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:15,815 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,815 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,815 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:15,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:15,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:15,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:15,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:15,816 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:15,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:15,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:15,817 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:15,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:15,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:15,820 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,820 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:15,820 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:15,820 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,820 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:15,820 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:15,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:15,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:15,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:15,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:15,820 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,820 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,820 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,820 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:15,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:15,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:15,820 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:15,821 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:15,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:15,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:15,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839035849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:15,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839035849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,921 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:15,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:15,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:15,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:15,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:15,924 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:15,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:15,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:15,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:15,925 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,925 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:15,927 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:15,927 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:15,927 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,927 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:15,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:15,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:15,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:15,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:15,928 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:15,928 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:15,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:15,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:15,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:15,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:15,929 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:15,929 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:15,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:15,929 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:15,929 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:15,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:15,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:15,929 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:15,930 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:15,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:15,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:15,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:15,937 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fb17f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:15,937 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:15,937 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,938 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,938 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:15,938 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:15,938 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:15,939 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,939 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:15,940 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,940 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:15,940 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:15,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:15,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:15,940 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:15,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:15,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:15,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:15,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:15,943 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:15,943 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:15,943 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:15,943 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:15,943 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:15,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,944 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:15,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:15,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:15,945 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:15,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:15,946 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:15,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:15,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,951 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,951 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:15,951 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:15,951 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:15,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:15,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:15,951 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:15,952 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,952 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:15,952 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:15,951 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:15,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:15,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:15,951 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:15,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:15,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:15,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:15,952 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:15,952 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:16,052 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:16,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:16,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:16,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:16,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:16,056 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:16,056 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:16,056 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:16,056 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:16,057 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:16,057 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:16,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:16,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839036054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:16,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839036055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,058 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:16,058 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:16,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:16,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:16,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:16,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:16,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:16,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:16,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:16,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:16,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:16,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:16,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:16,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:16,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:16,060 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:16,060 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:16,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:16,060 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:16,060 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:16,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:16,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:16,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:16,060 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:16,062 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:16,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:16,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:16,063 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:16,063 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:16,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:16,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:16,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:16,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:16,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:16,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25d7691e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:16,064 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:16,064 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:16,065 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:16,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:16,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:16,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:16,066 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:16,068 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,068 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:16,068 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:16,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:16,069 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:16,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:16,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:16,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:16,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:16,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:16,071 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:16,071 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:16,071 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:16,071 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:16,071 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:16,073 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:16,073 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:16,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:16,073 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:16,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:16,074 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,075 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:16,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,078 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:16,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:16,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:16,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:16,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:16,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:16,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:16,079 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:16,079 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:16,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:16,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:16,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:16,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:16,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:16,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:16,162 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/af5b1d5e585841a2a9058b55b843bb30 2023-03-26 13:56:16,172 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=454 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/b987176d581d4be092f8958fee226836 2023-03-26 13:56:16,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/e1aea2a8e93c46c8a2e7394f8e52de01 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/e1aea2a8e93c46c8a2e7394f8e52de01 2023-03-26 13:56:16,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/e1aea2a8e93c46c8a2e7394f8e52de01, entries=200, sequenceid=454, filesize=32.3 K 2023-03-26 13:56:16,179 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:16,179 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/af5b1d5e585841a2a9058b55b843bb30 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/af5b1d5e585841a2a9058b55b843bb30 2023-03-26 13:56:16,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:16,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:16,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:16,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:16,182 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:16,182 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:16,182 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:16,182 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:16,183 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:16,183 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:16,183 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/af5b1d5e585841a2a9058b55b843bb30, entries=150, sequenceid=454, filesize=11.9 K 2023-03-26 13:56:16,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/b987176d581d4be092f8958fee226836 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/b987176d581d4be092f8958fee226836 2023-03-26 13:56:16,189 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:16,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:16,189 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:16,189 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,189 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:16,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:16,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:16,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:16,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:16,191 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/b987176d581d4be092f8958fee226836, entries=150, sequenceid=454, filesize=11.9 K 2023-03-26 13:56:16,191 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~436.97 KB/447456, currentSize=26.84 KB/27480 for f904470a09c7c6a52bdab844746de048 in 456ms, sequenceid=454, compaction requested=true 2023-03-26 13:56:16,191 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:16,192 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:16,192 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:16,192 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:16,192 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-26 13:56:16,192 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-26 13:56:16,194 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 181546 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-26 13:56:16,194 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/A is initiating minor compaction (all files) 2023-03-26 13:56:16,194 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/A in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:16,194 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/6aee5876e3a246ed9628a2d3e8b747fd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/42fc1c46c1814651bc29473e0a7273db, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/439eafdc7d6a4360bb0a1a371681af6a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/e2c67069d15e4d4ebfff483b04823381, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/68eb1c0dade24dbb9eeed2f9de630e15, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/faa103eddba3441e8331cfd71d1851d7, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/e1aea2a8e93c46c8a2e7394f8e52de01] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=177.3 K 2023-03-26 13:56:16,194 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 79704 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-26 13:56:16,194 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/B is initiating minor compaction (all files) 2023-03-26 13:56:16,194 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/B in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:16,194 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/53eb9150507c4541953a4cfd1043eda0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/45c74fbd7d9540cfa34b36dc866f99b1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/d8eb7e553aef4cc89784022a8e1c3e3c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/91ad5684e0bb4e188f1e34b1b92d3a7c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/54f4bfff636845a19f1efc1dfbff70b8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/661c919c39bd4b048be6bd60de91b8dc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/af5b1d5e585841a2a9058b55b843bb30] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=77.8 K 2023-03-26 13:56:16,195 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 6aee5876e3a246ed9628a2d3e8b747fd, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679838970582 2023-03-26 13:56:16,195 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 53eb9150507c4541953a4cfd1043eda0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679838970582 2023-03-26 13:56:16,195 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 42fc1c46c1814651bc29473e0a7273db, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1679838970648 2023-03-26 13:56:16,195 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 45c74fbd7d9540cfa34b36dc866f99b1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1679838970648 2023-03-26 13:56:16,196 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 439eafdc7d6a4360bb0a1a371681af6a, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1679838971780 2023-03-26 13:56:16,196 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:16,196 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting d8eb7e553aef4cc89784022a8e1c3e3c, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1679838971780 2023-03-26 13:56:16,196 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:16,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:16,196 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e2c67069d15e4d4ebfff483b04823381, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1679838972801 2023-03-26 13:56:16,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:16,196 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:16,196 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 91ad5684e0bb4e188f1e34b1b92d3a7c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1679838972813 2023-03-26 13:56:16,197 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 68eb1c0dade24dbb9eeed2f9de630e15, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679838972925 2023-03-26 13:56:16,197 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting faa103eddba3441e8331cfd71d1851d7, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1679838973542 2023-03-26 13:56:16,197 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 54f4bfff636845a19f1efc1dfbff70b8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679838972925 2023-03-26 13:56:16,197 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e1aea2a8e93c46c8a2e7394f8e52de01, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679838975612 2023-03-26 13:56:16,198 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 661c919c39bd4b048be6bd60de91b8dc, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1679838973542 2023-03-26 13:56:16,198 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting af5b1d5e585841a2a9058b55b843bb30, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679838975612 2023-03-26 13:56:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:16,212 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:16,212 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:16,212 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:16,212 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-26 13:56:16,212 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:16,212 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:16,212 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:16,213 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:16,213 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:16,213 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:16,218 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#B#compaction#624 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:16,224 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#A#compaction#625 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:16,228 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/b75d47b92f6344dea23ffbdf9a4516b5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/b75d47b92f6344dea23ffbdf9a4516b5 2023-03-26 13:56:16,231 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in f904470a09c7c6a52bdab844746de048/B of f904470a09c7c6a52bdab844746de048 into b75d47b92f6344dea23ffbdf9a4516b5(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:16,232 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:16,232 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/B, priority=9, startTime=1679838976192; duration=0sec 2023-03-26 13:56:16,232 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:16,232 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-03-26 13:56:16,234 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 79704 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-03-26 13:56:16,234 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/C is initiating minor compaction (all files) 2023-03-26 13:56:16,234 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/C in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:16,234 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5cd34c17d1124e03892cf184b7ecfb5b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/66e4352320ce4b68b8383a950f941729, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/6e2f8f86530b44b292e5a106604e4c88, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/504a9b4a505641ad933963bae766d958, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/7e16d5539a624e918b35c44409d7112a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/592b3b19d7eb4e1090097d80a0c03c77, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/b987176d581d4be092f8958fee226836] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=77.8 K 2023-03-26 13:56:16,235 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 5cd34c17d1124e03892cf184b7ecfb5b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=121, earliestPutTs=1679838970582 2023-03-26 13:56:16,236 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 66e4352320ce4b68b8383a950f941729, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1679838970648 2023-03-26 13:56:16,236 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 6e2f8f86530b44b292e5a106604e4c88, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1679838971780 2023-03-26 13:56:16,236 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 504a9b4a505641ad933963bae766d958, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=182, earliestPutTs=1679838972813 2023-03-26 13:56:16,237 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 7e16d5539a624e918b35c44409d7112a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679838972925 2023-03-26 13:56:16,237 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 592b3b19d7eb4e1090097d80a0c03c77, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=244, earliestPutTs=1679838973542 2023-03-26 13:56:16,237 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting b987176d581d4be092f8958fee226836, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679838975612 2023-03-26 13:56:16,266 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#C#compaction#627 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:16,283 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/c637826cdead4dbc8f97a19019d69b20 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/c637826cdead4dbc8f97a19019d69b20 2023-03-26 13:56:16,286 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in f904470a09c7c6a52bdab844746de048/A of f904470a09c7c6a52bdab844746de048 into c637826cdead4dbc8f97a19019d69b20(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:16,287 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:16,287 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/A, priority=9, startTime=1679838976192; duration=0sec 2023-03-26 13:56:16,287 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:16,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:16,366 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:16,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679838976397, max timestamp=1679838976406Num uniques 859; 2023-03-26 13:56:16,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679838976397, max timestamp=1679838976406Num uniques 859; 2023-03-26 13:56:16,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:16,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:16,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838976397, max timestamp=1679838976406Num uniques 859; 2023-03-26 13:56:16,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:16,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679838976443, max timestamp=1679838976454Num uniques 1068; 2023-03-26 13:56:16,456 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-26 13:56:16,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679838976443, max timestamp=1679838976454Num uniques 1068; 2023-03-26 13:56:16,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679838976443, max timestamp=1679838976454Num uniques 1068; 2023-03-26 13:56:16,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-26 13:56:16,457 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-26 13:56:16,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679838976473, max timestamp=1679838976486Num uniques 1027; 2023-03-26 13:56:16,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-26 13:56:16,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679838976473, max timestamp=1679838976486Num uniques 1027; 2023-03-26 13:56:16,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-26 13:56:16,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679838976473, max timestamp=1679838976486Num uniques 1027; 2023-03-26 13:56:16,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-26 13:56:16,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679838976510, max timestamp=1679838976522Num uniques 1086; 2023-03-26 13:56:16,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-26 13:56:16,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679838976510, max timestamp=1679838976522Num uniques 1086; 2023-03-26 13:56:16,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679838976510, max timestamp=1679838976522Num uniques 1086; 2023-03-26 13:56:16,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-26 13:56:16,525 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-26 13:56:16,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679838976540, max timestamp=1679838976555Num uniques 1095; 2023-03-26 13:56:16,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679838976540, max timestamp=1679838976555Num uniques 1095; 2023-03-26 13:56:16,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-26 13:56:16,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-26 13:56:16,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679838976540, max timestamp=1679838976555Num uniques 1095; 2023-03-26 13:56:16,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=11220, suffix heap size=141848,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=245 2023-03-26 13:56:16,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:16,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839036586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679838976580, max timestamp=1679838976587Num uniques 1104; 2023-03-26 13:56:16,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679838976580, max timestamp=1679838976587Num uniques 1104; 2023-03-26 13:56:16,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-26 13:56:16,590 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-26 13:56:16,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679838976580, max timestamp=1679838976587Num uniques 1104; 2023-03-26 13:56:16,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-26 13:56:16,648 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679838976633, max timestamp=1679838976646Num uniques 1013; 2023-03-26 13:56:16,648 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679838976633, max timestamp=1679838976646Num uniques 1013; 2023-03-26 13:56:16,648 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679838976633, max timestamp=1679838976646Num uniques 1013; 2023-03-26 13:56:16,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-26 13:56:16,648 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-26 13:56:16,649 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-26 13:56:16,655 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326a6b0d40ee5df460eaaa365d76dde1703 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a6b0d40ee5df460eaaa365d76dde1703 2023-03-26 13:56:16,660 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=461, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/0918c56d03b54de7a4c4e966cba08d4f 2023-03-26 13:56:16,677 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/13537a9fdd044af0b15ceaab62013c4f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/13537a9fdd044af0b15ceaab62013c4f 2023-03-26 13:56:16,681 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in f904470a09c7c6a52bdab844746de048/C of f904470a09c7c6a52bdab844746de048 into 13537a9fdd044af0b15ceaab62013c4f(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:16,681 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:16,681 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/C, priority=9, startTime=1679838976192; duration=0sec 2023-03-26 13:56:16,681 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:16,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679838976676, max timestamp=1679838976709Num uniques 1022; 2023-03-26 13:56:16,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-26 13:56:16,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679838976676, max timestamp=1679838976709Num uniques 1022; 2023-03-26 13:56:16,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679838976676, max timestamp=1679838976709Num uniques 1022; 2023-03-26 13:56:16,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-26 13:56:16,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-03-26 13:56:16,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679838976722, max timestamp=1679838976736Num uniques 1031; 2023-03-26 13:56:16,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-26 13:56:16,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679838976722, max timestamp=1679838976736Num uniques 1031; 2023-03-26 13:56:16,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679838976722, max timestamp=1679838976736Num uniques 1031; 2023-03-26 13:56:16,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-26 13:56:16,739 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-26 13:56:16,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679838976753, max timestamp=1679838976762Num uniques 1090; 2023-03-26 13:56:16,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-26 13:56:16,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679838976753, max timestamp=1679838976762Num uniques 1090; 2023-03-26 13:56:16,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-26 13:56:16,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679838976753, max timestamp=1679838976762Num uniques 1090; 2023-03-26 13:56:16,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-26 13:56:16,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679838976796, max timestamp=1679838976802Num uniques 1049; 2023-03-26 13:56:16,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-03-26 13:56:16,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679838976796, max timestamp=1679838976802Num uniques 1049; 2023-03-26 13:56:16,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=11404, suffix heap size=137064,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=249 2023-03-26 13:56:16,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679838976797, max timestamp=1679838976802Num uniques 1049; 2023-03-26 13:56:16,806 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=6870, suffix heap size=137064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=150 2023-03-26 13:56:16,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679838976817, max timestamp=1679838976825Num uniques 1108; 2023-03-26 13:56:16,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-03-26 13:56:16,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679838976817, max timestamp=1679838976825Num uniques 1108; 2023-03-26 13:56:16,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679838976817, max timestamp=1679838976825Num uniques 1009; 2023-03-26 13:56:16,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-03-26 13:56:16,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46206, new segment data size=9520, suffix heap size=132904,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=208 2023-03-26 13:56:16,856 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679838976837, max timestamp=1679838976854Num uniques 1067; 2023-03-26 13:56:16,856 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-03-26 13:56:16,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679838976837, max timestamp=1679838976854Num uniques 1067; 2023-03-26 13:56:16,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679838976837, max timestamp=1679838976854Num uniques 1067; 2023-03-26 13:56:16,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-03-26 13:56:16,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=9932, suffix heap size=138936,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=217 2023-03-26 13:56:16,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679838976876, max timestamp=1679838976881Num uniques 1076; 2023-03-26 13:56:16,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-26 13:56:16,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679838976876, max timestamp=1679838976881Num uniques 1076; 2023-03-26 13:56:16,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-26 13:56:16,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679838976876, max timestamp=1679838976881Num uniques 1076; 2023-03-26 13:56:16,884 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-26 13:56:16,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679838976899, max timestamp=1679838976907Num uniques 1035; 2023-03-26 13:56:16,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-03-26 13:56:16,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679838976899, max timestamp=1679838976907Num uniques 1035; 2023-03-26 13:56:16,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-03-26 13:56:16,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1679838976899, max timestamp=1679838976907Num uniques 1035; 2023-03-26 13:56:16,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-03-26 13:56:16,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1679838976927, max timestamp=1679838976934Num uniques 1094; 2023-03-26 13:56:16,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1679838976927, max timestamp=1679838976934Num uniques 1094; 2023-03-26 13:56:16,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-03-26 13:56:16,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1679838976927, max timestamp=1679838976934Num uniques 1044; 2023-03-26 13:56:16,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-03-26 13:56:16,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-03-26 13:56:16,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:16,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839036965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:16,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1679838976955, max timestamp=1679838976965Num uniques 1103; 2023-03-26 13:56:16,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1679838976955, max timestamp=1679838976965Num uniques 1103; 2023-03-26 13:56:16,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=9295, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-03-26 13:56:16,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1679838976955, max timestamp=1679838976965Num uniques 1103; 2023-03-26 13:56:16,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=9295, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-03-26 13:56:16,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=9295, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-03-26 13:56:17,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1679838976984, max timestamp=1679838977010Num uniques 1062; 2023-03-26 13:56:17,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-03-26 13:56:17,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1679838976984, max timestamp=1679838977010Num uniques 1062; 2023-03-26 13:56:17,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-03-26 13:56:17,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1679838976984, max timestamp=1679838977010Num uniques 1062; 2023-03-26 13:56:17,013 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-03-26 13:56:17,053 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1679838977040, max timestamp=1679838977051Num uniques 1021; 2023-03-26 13:56:17,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-03-26 13:56:17,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1679838977040, max timestamp=1679838977051Num uniques 1021; 2023-03-26 13:56:17,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=171, cellSize=7826, totalHeapSize=18024, min timestamp=1679838977040, max timestamp=1679838977051Num uniques 1021; 2023-03-26 13:56:17,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-03-26 13:56:17,054 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=7826, suffix heap size=134152,new segment heap size=18024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=171 2023-03-26 13:56:17,068 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/1636dcf0a7214d689c1e06298a48a9cb 2023-03-26 13:56:17,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1679838977077, max timestamp=1679838977085Num uniques 1030; 2023-03-26 13:56:17,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-03-26 13:56:17,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1679838977077, max timestamp=1679838977085Num uniques 1030; 2023-03-26 13:56:17,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1679838977077, max timestamp=1679838977085Num uniques 1030; 2023-03-26 13:56:17,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-03-26 13:56:17,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47170, new segment data size=10530, suffix heap size=135088,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1030, new segment cells count=230 2023-03-26 13:56:17,114 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1679838977093, max timestamp=1679838977112Num uniques 1089; 2023-03-26 13:56:17,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=13234, suffix heap size=141224,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=289 2023-03-26 13:56:17,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1679838977093, max timestamp=1679838977112Num uniques 1089; 2023-03-26 13:56:17,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=13234, suffix heap size=141224,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=289 2023-03-26 13:56:17,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1679838977108, max timestamp=1679838977112Num uniques 1089; 2023-03-26 13:56:17,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=8654, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-03-26 13:56:17,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9058, totalHeapSize=20832, min timestamp=1679838977134, max timestamp=1679838977141Num uniques 1148; 2023-03-26 13:56:17,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=52568, new segment data size=9058, suffix heap size=147360,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=198 2023-03-26 13:56:17,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9058, totalHeapSize=20832, min timestamp=1679838977134, max timestamp=1679838977141Num uniques 1148; 2023-03-26 13:56:17,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52568, new segment data size=9058, suffix heap size=147360,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=198 2023-03-26 13:56:17,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9058, totalHeapSize=20832, min timestamp=1679838977134, max timestamp=1679838977141Num uniques 1048; 2023-03-26 13:56:17,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47988, new segment data size=9058, suffix heap size=136960,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=198 2023-03-26 13:56:17,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1679838977163, max timestamp=1679838977170Num uniques 1057; 2023-03-26 13:56:17,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=9475, suffix heap size=137896,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=207 2023-03-26 13:56:17,172 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1679838977163, max timestamp=1679838977170Num uniques 1057; 2023-03-26 13:56:17,174 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=9475, suffix heap size=137896,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=207 2023-03-26 13:56:17,173 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=207, cellSize=9475, totalHeapSize=21768, min timestamp=1679838977163, max timestamp=1679838977170Num uniques 1057; 2023-03-26 13:56:17,175 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=9475, suffix heap size=137896,new segment heap size=21768 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=207 2023-03-26 13:56:17,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1679838977191, max timestamp=1679838977201Num uniques 1066; 2023-03-26 13:56:17,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=9886, suffix heap size=138832,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=216 2023-03-26 13:56:17,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1679838977191, max timestamp=1679838977201Num uniques 1066; 2023-03-26 13:56:17,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=9886, suffix heap size=138832,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=216 2023-03-26 13:56:17,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1679838977191, max timestamp=1679838977201Num uniques 1066; 2023-03-26 13:56:17,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48816, new segment data size=9886, suffix heap size=138832,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1066, new segment cells count=216 2023-03-26 13:56:17,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8010, totalHeapSize=18440, min timestamp=1679838977218, max timestamp=1679838977226Num uniques 1075; 2023-03-26 13:56:17,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=8010, suffix heap size=139768,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=175 2023-03-26 13:56:17,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8010, totalHeapSize=18440, min timestamp=1679838977218, max timestamp=1679838977226Num uniques 1075; 2023-03-26 13:56:17,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=8010, suffix heap size=139768,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=175 2023-03-26 13:56:17,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679838977218, max timestamp=1679838977226Num uniques 1075; 2023-03-26 13:56:17,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=6870, suffix heap size=139768,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=150 2023-03-26 13:56:17,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1679838977239, max timestamp=1679838977252Num uniques 1034; 2023-03-26 13:56:17,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=10714, suffix heap size=135504,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=234 2023-03-26 13:56:17,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1679838977239, max timestamp=1679838977252Num uniques 1034; 2023-03-26 13:56:17,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1679838977239, max timestamp=1679838977252Num uniques 1009; 2023-03-26 13:56:17,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47354, new segment data size=10714, suffix heap size=135504,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=234 2023-03-26 13:56:17,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=10714, suffix heap size=132904,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=234 2023-03-26 13:56:17,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8838, totalHeapSize=20312, min timestamp=1679838977271, max timestamp=1679838977275Num uniques 1093; 2023-03-26 13:56:17,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8838, totalHeapSize=20312, min timestamp=1679838977271, max timestamp=1679838977275Num uniques 1093; 2023-03-26 13:56:17,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8838, totalHeapSize=20312, min timestamp=1679838977271, max timestamp=1679838977275Num uniques 1093; 2023-03-26 13:56:17,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=8838, suffix heap size=141640,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=193 2023-03-26 13:56:17,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=8838, suffix heap size=141640,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=193 2023-03-26 13:56:17,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=8838, suffix heap size=141640,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=193 2023-03-26 13:56:17,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1679838977296, max timestamp=1679838977302Num uniques 1052; 2023-03-26 13:56:17,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48180, new segment data size=6960, suffix heap size=137376,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1052, new segment cells count=152 2023-03-26 13:56:17,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1679838977296, max timestamp=1679838977302Num uniques 1052; 2023-03-26 13:56:17,304 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48180, new segment data size=6960, suffix heap size=137376,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1052, new segment cells count=152 2023-03-26 13:56:17,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=152, cellSize=6960, totalHeapSize=16048, min timestamp=1679838977296, max timestamp=1679838977302Num uniques 1052; 2023-03-26 13:56:17,305 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48180, new segment data size=6960, suffix heap size=137376,new segment heap size=16048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1052, new segment cells count=152 2023-03-26 13:56:17,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1679838977319, max timestamp=1679838977329Num uniques 1011; 2023-03-26 13:56:17,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1679838977319, max timestamp=1679838977329Num uniques 1011; 2023-03-26 13:56:17,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=7366, suffix heap size=133112,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=161 2023-03-26 13:56:17,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=7366, suffix heap size=133112,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=161 2023-03-26 13:56:17,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=161, cellSize=7366, totalHeapSize=16984, min timestamp=1679838977319, max timestamp=1679838977329Num uniques 1011; 2023-03-26 13:56:17,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46296, new segment data size=7366, suffix heap size=133112,new segment heap size=16984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1011, new segment cells count=161 2023-03-26 13:56:17,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7780, totalHeapSize=17920, min timestamp=1679838977347, max timestamp=1679838977356Num uniques 1020; 2023-03-26 13:56:17,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=7780, suffix heap size=134048,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=170 2023-03-26 13:56:17,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7780, totalHeapSize=17920, min timestamp=1679838977347, max timestamp=1679838977356Num uniques 1020; 2023-03-26 13:56:17,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=7780, suffix heap size=134048,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=170 2023-03-26 13:56:17,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=170, cellSize=7780, totalHeapSize=17920, min timestamp=1679838977347, max timestamp=1679838977356Num uniques 1020; 2023-03-26 13:56:17,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46710, new segment data size=7780, suffix heap size=134048,new segment heap size=17920 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1020, new segment cells count=170 2023-03-26 13:56:17,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1679838977375, max timestamp=1679838977379Num uniques 1029; 2023-03-26 13:56:17,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=8194, suffix heap size=134984,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=179 2023-03-26 13:56:17,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1679838977375, max timestamp=1679838977379Num uniques 1029; 2023-03-26 13:56:17,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=8194, suffix heap size=134984,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=179 2023-03-26 13:56:17,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1679838977375, max timestamp=1679838977379Num uniques 1029; 2023-03-26 13:56:17,381 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47124, new segment data size=8194, suffix heap size=134984,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1029, new segment cells count=179 2023-03-26 13:56:17,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1679838977395, max timestamp=1679838977403Num uniques 1038; 2023-03-26 13:56:17,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=10898, suffix heap size=135920,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=238 2023-03-26 13:56:17,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1679838977395, max timestamp=1679838977403Num uniques 1038; 2023-03-26 13:56:17,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=10898, suffix heap size=135920,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=238 2023-03-26 13:56:17,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=238, cellSize=10898, totalHeapSize=24992, min timestamp=1679838977395, max timestamp=1679838977403Num uniques 1038; 2023-03-26 13:56:17,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=10898, suffix heap size=135920,new segment heap size=24992 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=238 2023-03-26 13:56:17,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9022, totalHeapSize=20728, min timestamp=1679838977410, max timestamp=1679838977446Num uniques 1097; 2023-03-26 13:56:17,448 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=9022, suffix heap size=142056,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=197 2023-03-26 13:56:17,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1679838977410, max timestamp=1679838977446Num uniques 1097; 2023-03-26 13:56:17,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=247, cellSize=11312, totalHeapSize=25928, min timestamp=1679838977410, max timestamp=1679838977446Num uniques 1097; 2023-03-26 13:56:17,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=11312, suffix heap size=142056,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=247 2023-03-26 13:56:17,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50242, new segment data size=11312, suffix heap size=142056,new segment heap size=25928 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1097, new segment cells count=247 2023-03-26 13:56:17,479 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=461 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/ce45cc607feb447580e0f9415641f53f 2023-03-26 13:56:17,482 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/0918c56d03b54de7a4c4e966cba08d4f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/0918c56d03b54de7a4c4e966cba08d4f 2023-03-26 13:56:17,486 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/0918c56d03b54de7a4c4e966cba08d4f, entries=100, sequenceid=461, filesize=18.6 K 2023-03-26 13:56:17,486 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/1636dcf0a7214d689c1e06298a48a9cb as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1636dcf0a7214d689c1e06298a48a9cb 2023-03-26 13:56:17,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:17,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 430 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839037485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1679838977462, max timestamp=1679838977486Num uniques 1056; 2023-03-26 13:56:17,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=7140, suffix heap size=137792,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=156 2023-03-26 13:56:17,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1679838977462, max timestamp=1679838977486Num uniques 1106; 2023-03-26 13:56:17,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50650, new segment data size=7140, suffix heap size=142992,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1106, new segment cells count=156 2023-03-26 13:56:17,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=156, cellSize=7140, totalHeapSize=16464, min timestamp=1679838977462, max timestamp=1679838977486Num uniques 1106; 2023-03-26 13:56:17,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50650, new segment data size=7140, suffix heap size=142992,new segment heap size=16464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1106, new segment cells count=156 2023-03-26 13:56:17,491 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1636dcf0a7214d689c1e06298a48a9cb, entries=100, sequenceid=461, filesize=9.5 K 2023-03-26 13:56:17,492 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/ce45cc607feb447580e0f9415641f53f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/ce45cc607feb447580e0f9415641f53f 2023-03-26 13:56:17,495 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/ce45cc607feb447580e0f9415641f53f, entries=100, sequenceid=461, filesize=9.5 K 2023-03-26 13:56:17,496 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=40.25 KB/41220 for f904470a09c7c6a52bdab844746de048 in 1284ms, sequenceid=461, compaction requested=false 2023-03-26 13:56:17,496 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:17,496 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:17,496 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:17,496 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:17,496 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,496 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:17,496 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:17,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:17,498 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=53.67 KB heapSize=146.20 KB 2023-03-26 13:56:17,498 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:17,498 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:17,498 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:17,498 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:17,498 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:17,498 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:17,503 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:17,503 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:17,503 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,503 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:17,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:17,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18179fd6[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:17,505 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:17,505 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,506 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:17,507 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:17,507 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,507 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:17,508 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:17,508 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,508 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:17,508 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:17,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:17,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:17,512 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:17,512 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:17,512 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:17,512 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:17,512 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:17,513 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,513 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,513 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:17,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:17,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:17,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:17,515 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,516 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,519 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:17,519 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:17,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:17,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:17,519 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,519 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:17,519 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:17,519 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,519 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:17,519 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:17,519 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:17,519 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:17,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:17,519 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,519 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:17,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:17,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:17,528 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326ed9201018fe54bba98fd6c2743a18718 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ed9201018fe54bba98fd6c2743a18718 2023-03-26 13:56:17,534 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1056, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/29e006ab97b543f6b16e209667d7598e 2023-03-26 13:56:17,542 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1056 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/201036b34a0a4d37b385713c010b6416 2023-03-26 13:56:17,556 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1056 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/68a0cefd47c949cd992ef3611c2cbf90 2023-03-26 13:56:17,560 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/29e006ab97b543f6b16e209667d7598e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/29e006ab97b543f6b16e209667d7598e 2023-03-26 13:56:17,563 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/29e006ab97b543f6b16e209667d7598e, entries=200, sequenceid=1056, filesize=32.3 K 2023-03-26 13:56:17,564 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/201036b34a0a4d37b385713c010b6416 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/201036b34a0a4d37b385713c010b6416 2023-03-26 13:56:17,567 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/201036b34a0a4d37b385713c010b6416, entries=150, sequenceid=1056, filesize=11.9 K 2023-03-26 13:56:17,567 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/68a0cefd47c949cd992ef3611c2cbf90 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/68a0cefd47c949cd992ef3611c2cbf90 2023-03-26 13:56:17,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838977542, max timestamp=1679838977567Num uniques 859; 2023-03-26 13:56:17,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838977542, max timestamp=1679838977567Num uniques 859; 2023-03-26 13:56:17,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838977542, max timestamp=1679838977567Num uniques 859; 2023-03-26 13:56:17,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:17,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:17,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:17,571 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/68a0cefd47c949cd992ef3611c2cbf90, entries=150, sequenceid=1056, filesize=11.9 K 2023-03-26 13:56:17,572 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~146.16 KB/149664, currentSize=26.84 KB/27480 for f904470a09c7c6a52bdab844746de048 in 74ms, sequenceid=1056, compaction requested=true 2023-03-26 13:56:17,572 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:17,572 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:17,572 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:56:17,572 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:56:17,572 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:17,572 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:17,573 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78608 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:56:17,574 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/A is initiating minor compaction (all files) 2023-03-26 13:56:17,574 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/A in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:17,574 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/c637826cdead4dbc8f97a19019d69b20, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/0918c56d03b54de7a4c4e966cba08d4f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/29e006ab97b543f6b16e209667d7598e] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=76.8 K 2023-03-26 13:56:17,574 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34592 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:56:17,574 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting c637826cdead4dbc8f97a19019d69b20, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679838975612 2023-03-26 13:56:17,574 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/B is initiating minor compaction (all files) 2023-03-26 13:56:17,574 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/B in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:17,574 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/b75d47b92f6344dea23ffbdf9a4516b5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1636dcf0a7214d689c1e06298a48a9cb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/201036b34a0a4d37b385713c010b6416] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=33.8 K 2023-03-26 13:56:17,574 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 0918c56d03b54de7a4c4e966cba08d4f, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1679838975741 2023-03-26 13:56:17,575 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting b75d47b92f6344dea23ffbdf9a4516b5, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679838975612 2023-03-26 13:56:17,575 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 1636dcf0a7214d689c1e06298a48a9cb, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1679838975741 2023-03-26 13:56:17,575 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 29e006ab97b543f6b16e209667d7598e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1056, earliestPutTs=1679838977486 2023-03-26 13:56:17,575 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 201036b34a0a4d37b385713c010b6416, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1056, earliestPutTs=1679838977486 2023-03-26 13:56:17,582 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#B#compaction#634 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:17,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:17,583 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=53.67 KB heapSize=145.92 KB 2023-03-26 13:56:17,584 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:17,584 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:17,584 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:17,584 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:17,584 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:17,584 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:17,589 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#A#compaction#633 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:17,608 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326630c1a7002b84477840c9e4b9ee53f18 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326630c1a7002b84477840c9e4b9ee53f18 2023-03-26 13:56:17,615 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1081, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/12b61e62b99b469fae365cca2fe4bea2 2023-03-26 13:56:17,619 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/82bf935d2db94de7a07100b60b29137d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/82bf935d2db94de7a07100b60b29137d 2023-03-26 13:56:17,619 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:17,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:17,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:17,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:17,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:17,624 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:17,624 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:17,624 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:17,624 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:17,624 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,624 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:17,625 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f904470a09c7c6a52bdab844746de048/A of f904470a09c7c6a52bdab844746de048 into 82bf935d2db94de7a07100b60b29137d(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:17,625 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:17,625 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/A, priority=13, startTime=1679838977572; duration=0sec 2023-03-26 13:56:17,626 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:17,626 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:56:17,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838977593, max timestamp=1679838977624Num uniques 859; 2023-03-26 13:56:17,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:17,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838977593, max timestamp=1679838977624Num uniques 859; 2023-03-26 13:56:17,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:17,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838977593, max timestamp=1679838977624Num uniques 859; 2023-03-26 13:56:17,627 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:17,627 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:17,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:17,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:17,627 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,627 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34592 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:56:17,628 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/C is initiating minor compaction (all files) 2023-03-26 13:56:17,628 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/C in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:17,628 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/13537a9fdd044af0b15ceaab62013c4f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/ce45cc607feb447580e0f9415641f53f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/68a0cefd47c949cd992ef3611c2cbf90] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=33.8 K 2023-03-26 13:56:17,628 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,628 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:17,628 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 13537a9fdd044af0b15ceaab62013c4f, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=454, earliestPutTs=1679838975612 2023-03-26 13:56:17,628 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting ce45cc607feb447580e0f9415641f53f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=461, earliestPutTs=1679838975741 2023-03-26 13:56:17,629 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 68a0cefd47c949cd992ef3611c2cbf90, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1056, earliestPutTs=1679838977486 2023-03-26 13:56:17,629 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1081 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/af9b17e4c2d944ea8fdaa36e14644c5f 2023-03-26 13:56:17,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:17,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:17,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:17,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:17,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:17,636 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:17,636 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:17,636 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:17,637 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:17,637 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:17,637 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:17,637 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:17,637 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:17,637 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:17,637 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:17,637 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:17,637 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,637 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:17,637 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:17,638 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#C#compaction#638 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:17,639 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:17,639 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:17,640 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,640 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:17,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:17,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50caefd9[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:17,641 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:17,641 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,643 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,643 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,643 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:17,643 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:17,643 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:17,643 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:17,647 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:17,647 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,647 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:17,647 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:17,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:17,650 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:17,650 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:17,650 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:17,650 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:17,650 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:17,652 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1081 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/795b3597ae0e4b0ba5e6285e1cebb90b 2023-03-26 13:56:17,652 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,652 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,652 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:17,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:17,653 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:17,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:17,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:17,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,655 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:17,656 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/12b61e62b99b469fae365cca2fe4bea2 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/12b61e62b99b469fae365cca2fe4bea2 2023-03-26 13:56:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,659 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:17,659 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:17,659 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:17,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:17,659 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:17,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:17,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:17,659 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:17,659 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/12b61e62b99b469fae365cca2fe4bea2, entries=200, sequenceid=1081, filesize=32.3 K 2023-03-26 13:56:17,659 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:17,659 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:17,659 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,659 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,659 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,659 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:17,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:17,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:17,659 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:17,660 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/af9b17e4c2d944ea8fdaa36e14644c5f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/af9b17e4c2d944ea8fdaa36e14644c5f 2023-03-26 13:56:17,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:17,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 447 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839037658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,664 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/af9b17e4c2d944ea8fdaa36e14644c5f, entries=150, sequenceid=1081, filesize=11.9 K 2023-03-26 13:56:17,664 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/795b3597ae0e4b0ba5e6285e1cebb90b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/795b3597ae0e4b0ba5e6285e1cebb90b 2023-03-26 13:56:17,667 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/795b3597ae0e4b0ba5e6285e1cebb90b, entries=150, sequenceid=1081, filesize=11.9 K 2023-03-26 13:56:17,667 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~145.88 KB/149376, currentSize=134.18 KB/137400 for f904470a09c7c6a52bdab844746de048 in 84ms, sequenceid=1081, compaction requested=false 2023-03-26 13:56:17,667 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:17,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:17,669 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-26 13:56:17,670 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:17,670 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:17,670 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:17,670 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:17,670 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:17,670 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:17,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:17,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 597 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839037677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:17,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839037680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,685 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:17,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839037681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:17,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839037686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,690 DEBUG [Thread-4359] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4146 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., hostname=jenkins-hbase4.apache.org,42517,1679838814721, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:56:17,760 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:17,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:17,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:17,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:17,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 449 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839037763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:17,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:17,763 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:17,763 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:17,764 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:17,764 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:17,764 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,764 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:17,766 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:17,766 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:17,766 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:17,766 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:17,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:17,767 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:17,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:17,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:17,767 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:17,767 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:17,767 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:17,767 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:17,767 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:17,767 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:17,767 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:17,768 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:17,768 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:17,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:17,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:17,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:17,768 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:17,771 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,771 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:17,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:17,771 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,771 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:17,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:17,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d5348f6[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:17,773 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:17,773 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,774 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,775 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:17,775 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:17,775 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,775 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:17,775 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:17,776 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,776 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:17,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:17,776 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:17,776 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:17,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:17,779 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:17,779 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:17,779 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:17,779 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:17,779 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:17,781 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,781 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,781 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:17,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:17,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:17,782 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,783 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:17,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 599 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839037782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:17,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839037784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,787 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,787 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:17,788 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:17,788 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:17,788 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:17,788 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:17,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:17,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:17,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:17,788 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:17,788 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,788 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:17,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:17,788 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,788 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:17,788 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:17,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:17,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:17,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839037786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,888 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:17,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:17,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:17,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:17,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:17,892 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:17,892 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:17,892 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:17,892 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:17,893 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,893 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:17,894 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,894 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:17,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:17,894 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,894 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:17,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:17,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:17,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:17,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:17,896 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:17,896 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:17,896 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:17,896 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:17,896 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:17,896 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:17,896 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:17,896 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:17,896 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:17,896 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:17,896 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:17,897 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,897 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:17,897 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:17,898 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:17,898 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:17,899 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,899 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:17,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:17,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33d24bdb[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:17,900 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:17,900 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,901 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,901 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:17,901 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:17,901 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:17,901 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,901 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:17,904 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:17,904 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,904 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:17,904 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:17,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:17,906 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:17,906 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:17,906 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:17,907 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:17,907 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:17,908 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,908 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:17,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,908 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:17,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:17,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:17,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:17,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:17,909 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:17,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:17,910 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:17,912 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:17,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:17,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:17,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:17,913 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:17,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:17,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:17,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:17,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:17,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,913 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:17,913 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:17,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:17,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:17,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:17,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:17,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:17,967 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:17,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 451 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839037964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:17,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 601 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839037987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:17,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839037987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,993 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/bac998b9ce2f45a3bd58aa843d1b43be as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/bac998b9ce2f45a3bd58aa843d1b43be 2023-03-26 13:56:17,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:17,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839037991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:17,997 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f904470a09c7c6a52bdab844746de048/B of f904470a09c7c6a52bdab844746de048 into bac998b9ce2f45a3bd58aa843d1b43be(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:17,997 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:17,997 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/B, priority=13, startTime=1679838977572; duration=0sec 2023-03-26 13:56:17,997 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:18,013 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:18,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:18,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:18,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:18,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:18,016 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:18,016 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:18,016 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:18,016 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:18,017 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:18,017 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:18,018 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:18,018 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:18,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:18,018 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,018 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:18,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:18,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:18,019 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:18,019 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:18,019 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:18,019 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:18,019 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:18,019 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:18,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:18,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:18,020 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:18,020 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:18,020 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:18,020 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:18,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:18,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:18,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:18,020 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:18,024 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:18,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:18,024 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:18,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:18,024 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:18,024 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:18,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:18,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:18,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:18,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:18,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@171f8b58[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:18,026 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:18,026 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:18,027 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:18,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:18,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:18,027 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:18,027 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:18,027 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:18,027 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,028 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:18,028 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,028 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:18,029 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:18,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:18,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:18,029 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:18,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:18,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:18,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:18,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:18,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:18,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:18,031 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:18,032 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:18,032 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:18,032 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:18,032 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:18,036 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:18,036 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:18,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:18,036 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:18,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:18,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:18,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:18,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:18,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:18,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:18,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:18,037 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:18,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:18,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:18,038 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:18,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:18,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:18,040 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,040 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:18,040 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:18,040 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,040 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:18,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:18,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:18,040 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:18,041 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:18,040 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:18,041 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:18,040 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:18,041 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:18,041 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:18,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:18,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:18,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:18,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:18,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:18,066 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/684a473d246b4fbb8b3f9d0996d32e43 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/684a473d246b4fbb8b3f9d0996d32e43 2023-03-26 13:56:18,070 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f904470a09c7c6a52bdab844746de048/C of f904470a09c7c6a52bdab844746de048 into 684a473d246b4fbb8b3f9d0996d32e43(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:18,070 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:18,070 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/C, priority=13, startTime=1679838977572; duration=0sec 2023-03-26 13:56:18,070 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:18,090 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326d0157d9a2df24945aed2161c860337ae to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326d0157d9a2df24945aed2161c860337ae 2023-03-26 13:56:18,098 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1119, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/c719f4fcd9fd456587bc48b7c6cef579 2023-03-26 13:56:18,109 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1119 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/539fea6ceb3446bcb01cdada2536760f 2023-03-26 13:56:18,130 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1119 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/f5c9a26cddd24fbf9950540e374697ed 2023-03-26 13:56:18,134 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/c719f4fcd9fd456587bc48b7c6cef579 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/c719f4fcd9fd456587bc48b7c6cef579 2023-03-26 13:56:18,137 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/c719f4fcd9fd456587bc48b7c6cef579, entries=200, sequenceid=1119, filesize=32.3 K 2023-03-26 13:56:18,137 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/539fea6ceb3446bcb01cdada2536760f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/539fea6ceb3446bcb01cdada2536760f 2023-03-26 13:56:18,140 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/539fea6ceb3446bcb01cdada2536760f, entries=150, sequenceid=1119, filesize=11.9 K 2023-03-26 13:56:18,141 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/f5c9a26cddd24fbf9950540e374697ed as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/f5c9a26cddd24fbf9950540e374697ed 2023-03-26 13:56:18,141 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:18,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:18,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:18,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:18,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:18,144 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:18,144 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:18,144 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:18,144 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:18,144 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/f5c9a26cddd24fbf9950540e374697ed, entries=150, sequenceid=1119, filesize=11.9 K 2023-03-26 13:56:18,145 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:18,145 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:18,145 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for f904470a09c7c6a52bdab844746de048 in 476ms, sequenceid=1119, compaction requested=true 2023-03-26 13:56:18,145 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:18,145 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:18,145 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:56:18,146 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:56:18,145 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:18,146 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:18,146 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:18,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:18,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:18,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:18,147 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:56:18,146 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,147 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/B is initiating minor compaction (all files) 2023-03-26 13:56:18,147 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/B in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:18,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:18,147 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92798 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:56:18,147 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,147 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:18,147 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/bac998b9ce2f45a3bd58aa843d1b43be, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/af9b17e4c2d944ea8fdaa36e14644c5f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/539fea6ceb3446bcb01cdada2536760f] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=36.3 K 2023-03-26 13:56:18,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:18,147 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/A is initiating minor compaction (all files) 2023-03-26 13:56:18,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:18,148 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/A in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:18,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:18,148 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting bac998b9ce2f45a3bd58aa843d1b43be, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1056, earliestPutTs=1679838977486 2023-03-26 13:56:18,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:18,148 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/82bf935d2db94de7a07100b60b29137d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/12b61e62b99b469fae365cca2fe4bea2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/c719f4fcd9fd456587bc48b7c6cef579] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=90.6 K 2023-03-26 13:56:18,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:18,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:18,148 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting af9b17e4c2d944ea8fdaa36e14644c5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1081, earliestPutTs=1679838977542 2023-03-26 13:56:18,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:18,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:18,148 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 82bf935d2db94de7a07100b60b29137d, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=1056, earliestPutTs=1679838977486 2023-03-26 13:56:18,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:18,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:18,149 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 539fea6ceb3446bcb01cdada2536760f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1119, earliestPutTs=1679838977656 2023-03-26 13:56:18,148 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:18,149 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:18,149 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 12b61e62b99b469fae365cca2fe4bea2, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1081, earliestPutTs=1679838977542 2023-03-26 13:56:18,149 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:56:18,149 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:18,149 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:18,149 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:18,149 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:18,149 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:18,149 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:18,149 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting c719f4fcd9fd456587bc48b7c6cef579, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1119, earliestPutTs=1679838977641 2023-03-26 13:56:18,157 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#B#compaction#643 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:18,165 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#A#compaction#644 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:18,190 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/4590cff7c622414b8752dfb73ba71a58 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/4590cff7c622414b8752dfb73ba71a58 2023-03-26 13:56:18,195 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f904470a09c7c6a52bdab844746de048/A of f904470a09c7c6a52bdab844746de048 into 4590cff7c622414b8752dfb73ba71a58(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:18,195 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:18,195 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/A, priority=13, startTime=1679838978145; duration=0sec 2023-03-26 13:56:18,195 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:18,195 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:56:18,196 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:56:18,196 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/C is initiating minor compaction (all files) 2023-03-26 13:56:18,196 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/C in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:18,196 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/684a473d246b4fbb8b3f9d0996d32e43, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/795b3597ae0e4b0ba5e6285e1cebb90b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/f5c9a26cddd24fbf9950540e374697ed] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=36.3 K 2023-03-26 13:56:18,196 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 684a473d246b4fbb8b3f9d0996d32e43, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=1056, earliestPutTs=1679838977486 2023-03-26 13:56:18,197 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 795b3597ae0e4b0ba5e6285e1cebb90b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1081, earliestPutTs=1679838977542 2023-03-26 13:56:18,197 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting f5c9a26cddd24fbf9950540e374697ed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1119, earliestPutTs=1679838977656 2023-03-26 13:56:18,202 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#C#compaction#645 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:18,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:18,272 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:18,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838978297, max timestamp=1679838978301Num uniques 859; 2023-03-26 13:56:18,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:18,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838978297, max timestamp=1679838978301Num uniques 859; 2023-03-26 13:56:18,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:18,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838978297, max timestamp=1679838978301Num uniques 859; 2023-03-26 13:56:18,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:18,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:18,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 468 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839038337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:18,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839038339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:18,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46358 deadline: 1679839038340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:18,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 610 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839038340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:18,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 470 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839038441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:18,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839038444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:18,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839038445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:18,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 612 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839038445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,570 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303268f73597ee445476684a760a19f96aec8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303268f73597ee445476684a760a19f96aec8 2023-03-26 13:56:18,571 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/b6410a487db54cbdaeefedc05669dde2 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/b6410a487db54cbdaeefedc05669dde2 2023-03-26 13:56:18,575 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1131, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/8daf3bd1b5454ec3a28da6bbaf842336 2023-03-26 13:56:18,577 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f904470a09c7c6a52bdab844746de048/B of f904470a09c7c6a52bdab844746de048 into b6410a487db54cbdaeefedc05669dde2(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:18,577 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:18,577 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/B, priority=13, startTime=1679838978145; duration=0sec 2023-03-26 13:56:18,577 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:18,623 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/15cc4feea2304544953a476e27e1f90a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/15cc4feea2304544953a476e27e1f90a 2023-03-26 13:56:18,627 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f904470a09c7c6a52bdab844746de048/C of f904470a09c7c6a52bdab844746de048 into 15cc4feea2304544953a476e27e1f90a(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:18,627 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:18,627 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/C, priority=13, startTime=1679838978146; duration=0sec 2023-03-26 13:56:18,627 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:18,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:18,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 472 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839038643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:18,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839038647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:18,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839038647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:18,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 614 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839038647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:18,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 474 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839038948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:18,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839038951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:18,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:18,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839038951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 616 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839038952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:18,984 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1131 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/3731a9ed513d43f3958984221b023a9a 2023-03-26 13:56:19,395 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1131 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/aaef2502f97444d7bd46de64dca2be80 2023-03-26 13:56:19,407 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/8daf3bd1b5454ec3a28da6bbaf842336 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/8daf3bd1b5454ec3a28da6bbaf842336 2023-03-26 13:56:19,411 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/8daf3bd1b5454ec3a28da6bbaf842336, entries=150, sequenceid=1131, filesize=25.4 K 2023-03-26 13:56:19,412 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/3731a9ed513d43f3958984221b023a9a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/3731a9ed513d43f3958984221b023a9a 2023-03-26 13:56:19,416 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/3731a9ed513d43f3958984221b023a9a, entries=150, sequenceid=1131, filesize=11.9 K 2023-03-26 13:56:19,418 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/aaef2502f97444d7bd46de64dca2be80 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/aaef2502f97444d7bd46de64dca2be80 2023-03-26 13:56:19,425 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/aaef2502f97444d7bd46de64dca2be80, entries=150, sequenceid=1131, filesize=11.9 K 2023-03-26 13:56:19,426 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f904470a09c7c6a52bdab844746de048 in 1277ms, sequenceid=1131, compaction requested=false 2023-03-26 13:56:19,426 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:19,426 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:19,426 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:19,426 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:19,426 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,426 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:19,426 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:19,428 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,428 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:19,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:19,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:19,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:19,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ac1a45[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:19,430 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:19,430 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,431 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,432 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:19,432 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:19,432 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:19,432 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,432 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:19,433 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:19,433 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,433 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:19,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:19,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:19,433 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:19,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:19,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:19,436 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:19,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:19,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:19,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:19,436 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:19,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:19,437 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:19,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:19,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:19,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,439 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:19,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:19,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:19,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:19,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:19,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:19,441 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,442 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,442 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,442 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:19,441 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:19,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:19,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:19,442 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,442 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:19,442 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:19,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:19,456 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-26 13:56:19,456 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:19,456 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:19,456 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:19,456 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:19,456 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:19,456 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:19,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:19,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 619 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839039461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:19,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 479 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839039464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,465 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:19,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839039464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:19,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839039468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,480 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326b7e1a5203ec24c76895bb59588956dba to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326b7e1a5203ec24c76895bb59588956dba 2023-03-26 13:56:19,485 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1171, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/478de32ff85f48b49eeb1b1335a1c783 2023-03-26 13:56:19,492 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1171 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/808770745f8b4d08a9afd1ed9d11e1e0 2023-03-26 13:56:19,524 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1171 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/c8e812fc39c54d83816d095dd99c86ab 2023-03-26 13:56:19,529 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/478de32ff85f48b49eeb1b1335a1c783 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/478de32ff85f48b49eeb1b1335a1c783 2023-03-26 13:56:19,533 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/478de32ff85f48b49eeb1b1335a1c783, entries=200, sequenceid=1171, filesize=32.3 K 2023-03-26 13:56:19,534 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/808770745f8b4d08a9afd1ed9d11e1e0 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/808770745f8b4d08a9afd1ed9d11e1e0 2023-03-26 13:56:19,538 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/808770745f8b4d08a9afd1ed9d11e1e0, entries=150, sequenceid=1171, filesize=11.9 K 2023-03-26 13:56:19,538 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/c8e812fc39c54d83816d095dd99c86ab as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/c8e812fc39c54d83816d095dd99c86ab 2023-03-26 13:56:19,541 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/c8e812fc39c54d83816d095dd99c86ab, entries=150, sequenceid=1171, filesize=11.9 K 2023-03-26 13:56:19,542 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for f904470a09c7c6a52bdab844746de048 in 86ms, sequenceid=1171, compaction requested=true 2023-03-26 13:56:19,542 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:19,542 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:19,542 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:19,542 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-26 13:56:19,542 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-26 13:56:19,543 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:56:19,543 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:56:19,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:19,544 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85860 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:56:19,544 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:56:19,544 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/A is initiating minor compaction (all files) 2023-03-26 13:56:19,544 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/C is initiating minor compaction (all files) 2023-03-26 13:56:19,544 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/A in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:19,544 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/C in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:19,544 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/4590cff7c622414b8752dfb73ba71a58, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/8daf3bd1b5454ec3a28da6bbaf842336, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/478de32ff85f48b49eeb1b1335a1c783] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=83.8 K 2023-03-26 13:56:19,544 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/15cc4feea2304544953a476e27e1f90a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/aaef2502f97444d7bd46de64dca2be80, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/c8e812fc39c54d83816d095dd99c86ab] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=36.4 K 2023-03-26 13:56:19,545 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 15cc4feea2304544953a476e27e1f90a, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=1119, earliestPutTs=1679838977656 2023-03-26 13:56:19,545 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 4590cff7c622414b8752dfb73ba71a58, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=1119, earliestPutTs=1679838977656 2023-03-26 13:56:19,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:19,545 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting aaef2502f97444d7bd46de64dca2be80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1131, earliestPutTs=1679838977676 2023-03-26 13:56:19,545 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 8daf3bd1b5454ec3a28da6bbaf842336, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1131, earliestPutTs=1679838977676 2023-03-26 13:56:19,546 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 478de32ff85f48b49eeb1b1335a1c783, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1171, earliestPutTs=1679838978334 2023-03-26 13:56:19,546 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting c8e812fc39c54d83816d095dd99c86ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1171, earliestPutTs=1679838978334 2023-03-26 13:56:19,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:19,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:19,547 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:19,547 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:19,547 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:19,547 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:19,548 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,548 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,551 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:19,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:19,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:19,551 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,552 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,552 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:19,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:19,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:19,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:19,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:19,553 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:19,553 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:19,553 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:19,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:19,567 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 13:56:19,567 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:19,568 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:19,568 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:19,568 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:19,568 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:19,568 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:19,568 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:19,568 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:19,568 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:19,568 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:19,568 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:19,568 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:19,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:19,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:19,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:19,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:19,569 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#C#compaction#652 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:19,570 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,570 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:19,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:19,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,571 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:19,571 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#A#compaction#651 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:19,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:19,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:19,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:19,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:19,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@270ff45e[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:19,573 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:19,573 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,574 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,574 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:19,574 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:19,574 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:19,574 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,574 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:19,576 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,576 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:19,576 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:19,576 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:19,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:19,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:19,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:19,579 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:19,579 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:19,579 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:19,579 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:19,579 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:19,582 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,582 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,582 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:19,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:19,583 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:19,584 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,586 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:19,586 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:19,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:19,586 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:19,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:19,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:19,586 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,586 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,586 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,586 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,586 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,586 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,586 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:19,586 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:19,586 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:19,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:19,594 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/1308f8b1e31a473abdeb636f10e5c77c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/1308f8b1e31a473abdeb636f10e5c77c 2023-03-26 13:56:19,602 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f904470a09c7c6a52bdab844746de048/C of f904470a09c7c6a52bdab844746de048 into 1308f8b1e31a473abdeb636f10e5c77c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:19,602 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:19,602 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/C, priority=13, startTime=1679838979542; duration=0sec 2023-03-26 13:56:19,602 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:19,602 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:56:19,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1679838979568, max timestamp=1679838979601Num uniques 859; 2023-03-26 13:56:19,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-03-26 13:56:19,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838979590, max timestamp=1679838979601Num uniques 859; 2023-03-26 13:56:19,603 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:19,603 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:56:19,604 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/B is initiating minor compaction (all files) 2023-03-26 13:56:19,604 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/B in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:19,604 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/b6410a487db54cbdaeefedc05669dde2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/3731a9ed513d43f3958984221b023a9a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/808770745f8b4d08a9afd1ed9d11e1e0] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=36.4 K 2023-03-26 13:56:19,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838979590, max timestamp=1679838979601Num uniques 859; 2023-03-26 13:56:19,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:19,605 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting b6410a487db54cbdaeefedc05669dde2, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=1119, earliestPutTs=1679838977656 2023-03-26 13:56:19,605 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 3731a9ed513d43f3958984221b023a9a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1131, earliestPutTs=1679838977676 2023-03-26 13:56:19,606 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 808770745f8b4d08a9afd1ed9d11e1e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1171, earliestPutTs=1679838978334 2023-03-26 13:56:19,608 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/54da4f30389e44f8a068b8b1d23a8aed as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/54da4f30389e44f8a068b8b1d23a8aed 2023-03-26 13:56:19,612 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#B#compaction#654 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:19,612 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f904470a09c7c6a52bdab844746de048/A of f904470a09c7c6a52bdab844746de048 into 54da4f30389e44f8a068b8b1d23a8aed(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:19,613 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:19,613 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/A, priority=13, startTime=1679838979542; duration=0sec 2023-03-26 13:56:19,613 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:19,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:19,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 488 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839039616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:19,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839039617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:19,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 629 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839039617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:19,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46350 deadline: 1679839039619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,622 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/301d14ff980045e78380989fb48c898b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/301d14ff980045e78380989fb48c898b 2023-03-26 13:56:19,626 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f904470a09c7c6a52bdab844746de048/B of f904470a09c7c6a52bdab844746de048 into 301d14ff980045e78380989fb48c898b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:19,626 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:19,627 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/B, priority=13, startTime=1679838979542; duration=0sec 2023-03-26 13:56:19,627 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:19,686 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:19,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:19,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:19,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:19,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:19,690 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:19,690 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:19,690 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:19,690 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:19,691 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,691 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,692 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:19,693 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,693 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,693 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:19,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:19,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:19,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:19,694 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:19,694 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:19,694 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:19,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:19,695 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:19,695 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:19,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:19,695 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:19,695 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:19,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:19,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:19,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:19,695 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:19,696 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,696 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:19,697 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,697 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:19,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:19,698 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:19,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@622fff7[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:19,698 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,701 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:19,701 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:19,701 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:19,702 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,702 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:19,703 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,703 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:19,703 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:19,703 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:19,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:19,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:19,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:19,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:19,707 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:19,707 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:19,707 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:19,707 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:19,707 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:19,709 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,709 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,709 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:19,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:19,709 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:19,710 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:19,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:19,717 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:19,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:19,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:19,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:19,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:19,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:19,718 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:19,718 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:19,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,718 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,718 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:19,718 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,718 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:19,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 490 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839039722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:19,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:19,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839039722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 631 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839039722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:19,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839039722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,818 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:19,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:19,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:19,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:19,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:19,821 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:19,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:19,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:19,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:19,822 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,822 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,823 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,823 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:19,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:19,823 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,823 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:19,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:19,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:19,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:19,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:19,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:19,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:19,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:19,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:19,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:19,825 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:19,825 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:19,825 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:19,825 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:19,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:19,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:19,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:19,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:19,827 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,827 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:19,827 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,827 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:19,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:19,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:19,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d8cf120[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:19,829 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:19,829 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,830 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:19,830 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:19,830 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,830 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:19,833 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:19,833 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:19,833 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:19,833 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:19,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:19,835 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:19,835 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:19,835 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:19,835 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:19,835 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:19,837 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,837 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:19,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,837 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:19,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:19,838 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:19,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,839 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:19,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:19,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:19,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:19,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,842 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:19,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:19,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:19,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:19,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:19,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,842 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:19,842 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:19,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:19,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 633 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839039925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:19,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:19,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 492 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839039926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839039926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:19,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839039926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,943 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:19,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:19,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:19,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:19,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:19,946 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:19,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:19,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:19,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:19,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,948 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:19,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:19,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:19,949 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,950 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,950 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,950 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:19,950 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:19,950 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:19,950 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:19,950 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:19,950 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:19,950 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:19,951 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:19,951 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:19,951 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,951 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:19,951 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:19,953 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,953 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:19,954 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,954 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:19,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:19,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:19,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:19,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d82a0e6[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:19,955 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:19,956 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,956 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:19,957 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:19,957 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,957 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:19,958 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:19,958 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,958 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:19,958 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:19,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:19,960 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:19,960 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:19,961 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:19,961 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:19,961 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:19,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:19,963 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,963 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:19,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,963 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:19,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:19,963 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,964 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:19,967 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,967 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,967 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:19,967 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:19,967 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,967 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:19,967 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:19,967 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:19,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:19,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:19,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:19,968 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:19,968 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:19,968 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:19,968 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:19,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:19,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:19,994 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303269127725d10ec4457be7c2fe3bd79ce3f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303269127725d10ec4457be7c2fe3bd79ce3f 2023-03-26 13:56:20,068 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:20,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:20,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:20,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:20,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:20,072 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:20,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:20,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:20,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:20,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,073 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:20,074 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:20,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,074 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,074 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:20,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:20,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:20,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:20,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:20,075 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:20,075 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:20,075 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:20,075 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:20,075 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:20,075 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:20,075 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:20,076 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:20,076 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:20,076 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:20,078 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,078 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:20,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:20,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:20,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45f71a0c[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:20,080 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:20,080 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,081 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,082 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:20,082 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:20,082 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:20,082 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,082 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:20,083 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:20,083 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:20,083 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:20,083 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:20,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:20,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:20,085 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:20,085 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:20,085 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:20,085 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:20,086 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:20,088 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,088 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:20,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,088 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:20,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:20,089 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,090 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,092 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,092 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,092 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,092 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,092 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,092 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,092 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,092 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:20,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:20,093 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:20,093 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:20,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:20,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:20,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:20,093 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:20,093 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:20,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:20,193 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:20,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:20,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:20,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:20,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:20,196 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:20,196 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:20,196 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:20,196 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:20,197 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,197 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,201 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:20,201 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:20,201 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,201 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:20,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:20,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:20,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:20,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:20,202 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:20,202 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:20,203 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:20,203 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:20,203 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:20,203 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:20,203 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:20,203 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:20,203 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:20,203 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:20,203 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:20,203 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,203 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:20,203 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:20,204 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,204 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:20,205 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,205 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:20,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:20,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:20,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@543b8802[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:20,206 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:20,206 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,208 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:20,208 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:20,208 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:20,208 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,208 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:20,211 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:20,211 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:20,211 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,211 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:20,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:20,213 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:20,213 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:20,213 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:20,213 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:20,213 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:20,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:20,214 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:20,215 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:20,216 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,218 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:20,218 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:20,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:20,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:20,218 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,218 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:20,219 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:20,218 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:20,219 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:20,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:20,218 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,219 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,219 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,219 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,219 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:20,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:20,231 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:20,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 635 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839040229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839040229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:20,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839040230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:20,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 494 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839040231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,319 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:20,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:20,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:20,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:20,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:20,322 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:20,322 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:20,322 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:20,322 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:20,322 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,322 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,323 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,323 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:20,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:20,324 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,324 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:20,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,325 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:20,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,325 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:20,325 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:20,325 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:20,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:20,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:20,325 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:20,326 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:20,326 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:20,326 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:20,326 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:20,326 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:20,329 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,329 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:20,329 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,329 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:20,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:20,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:20,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5eb47654[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:20,331 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:20,331 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,332 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,332 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:20,332 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:20,332 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:20,332 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,333 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:20,334 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,334 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:20,334 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:20,334 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:20,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:20,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:20,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:20,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:20,337 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:20,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:20,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:20,337 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:20,337 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:20,339 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,339 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:20,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,339 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:20,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:20,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:20,340 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,341 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:20,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,344 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:20,344 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:20,344 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:20,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:20,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:20,344 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,345 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:20,345 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:20,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:20,345 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,345 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,345 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,345 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,345 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,345 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:20,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:20,345 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:20,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:20,399 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1182, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/04a2b7185a5c4727a42c62e84e1916ad 2023-03-26 13:56:20,408 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1182 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/1d4e38503fc74fdf89b8ff9a30f61d93 2023-03-26 13:56:20,418 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1182 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/a7fb8a70234a4cd08f7c84179546b1ea 2023-03-26 13:56:20,422 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/04a2b7185a5c4727a42c62e84e1916ad as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/04a2b7185a5c4727a42c62e84e1916ad 2023-03-26 13:56:20,425 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/04a2b7185a5c4727a42c62e84e1916ad, entries=150, sequenceid=1182, filesize=25.4 K 2023-03-26 13:56:20,426 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/1d4e38503fc74fdf89b8ff9a30f61d93 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1d4e38503fc74fdf89b8ff9a30f61d93 2023-03-26 13:56:20,431 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1d4e38503fc74fdf89b8ff9a30f61d93, entries=150, sequenceid=1182, filesize=11.9 K 2023-03-26 13:56:20,431 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/a7fb8a70234a4cd08f7c84179546b1ea as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/a7fb8a70234a4cd08f7c84179546b1ea 2023-03-26 13:56:20,435 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/a7fb8a70234a4cd08f7c84179546b1ea, entries=150, sequenceid=1182, filesize=11.9 K 2023-03-26 13:56:20,435 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=136.42 KB/139690 for f904470a09c7c6a52bdab844746de048 in 868ms, sequenceid=1182, compaction requested=false 2023-03-26 13:56:20,435 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:20,445 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:20,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:20,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:20,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:20,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:20,449 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:20,449 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:20,449 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:20,449 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:20,450 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,450 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,451 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,451 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:20,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,452 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,452 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:20,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,452 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:20,453 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:20,453 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:20,453 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:20,453 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:20,453 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:20,453 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:20,453 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:20,453 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:20,453 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:20,453 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=136.42 KB heapSize=366.63 KB 2023-03-26 13:56:20,454 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:20,454 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:20,454 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:20,454 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:20,454 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:20,454 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:20,477 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326555a8924acb546b4972ff48d7c53c7f5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326555a8924acb546b4972ff48d7c53c7f5 2023-03-26 13:56:20,495 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1217, memsize=64.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/202cbfcaebca444cb921ad780065d964 2023-03-26 13:56:20,505 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1217 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/074027c947aa4867b7a6cb93103612ef 2023-03-26 13:56:20,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:20,735 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:20,809 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:20,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839040794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,811 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:20,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839040809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:20,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 639 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839040809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:20,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 498 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839040809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:20,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839040910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:20,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 641 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839040913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:20,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839040913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:20,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 500 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839040913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,915 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1217 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/246e7b75270c45878fda2544e269e9c5 2023-03-26 13:56:20,919 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/202cbfcaebca444cb921ad780065d964 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/202cbfcaebca444cb921ad780065d964 2023-03-26 13:56:20,922 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/202cbfcaebca444cb921ad780065d964, entries=150, sequenceid=1217, filesize=25.4 K 2023-03-26 13:56:20,923 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/074027c947aa4867b7a6cb93103612ef as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/074027c947aa4867b7a6cb93103612ef 2023-03-26 13:56:20,927 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/074027c947aa4867b7a6cb93103612ef, entries=150, sequenceid=1217, filesize=11.9 K 2023-03-26 13:56:20,927 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/246e7b75270c45878fda2544e269e9c5 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/246e7b75270c45878fda2544e269e9c5 2023-03-26 13:56:20,930 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/246e7b75270c45878fda2544e269e9c5, entries=150, sequenceid=1217, filesize=11.9 K 2023-03-26 13:56:20,931 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.42 KB/139690, heapSize ~366.58 KB/375376, currentSize=67.09 KB/68700 for f904470a09c7c6a52bdab844746de048 in 478ms, sequenceid=1217, compaction requested=true 2023-03-26 13:56:20,931 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:20,931 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:20,931 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:20,931 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:20,931 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,931 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:20,931 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:20,932 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,932 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:20,933 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,933 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:20,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:20,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:20,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56b978f4[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:20,934 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:20,934 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:20,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:20,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:20,935 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,935 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:20,938 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:20,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:20,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:20,938 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:20,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:20,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:20,940 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:20,940 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:20,940 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:20,940 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:20,940 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:20,942 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,942 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:20,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,942 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:20,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:20,942 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:20,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:20,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,943 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:20,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:20,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:20,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:20,945 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:20,946 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:20,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:20,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:20,946 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:20,946 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,946 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:20,946 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:20,946 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:20,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:20,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:20,946 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:20,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:21,046 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:21,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:21,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:21,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:21,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:21,050 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:21,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:21,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:21,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:21,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:21,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:21,052 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,052 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:21,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:21,052 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,052 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:21,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:21,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:21,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:21,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:21,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:21,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:21,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:21,053 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:21,053 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:21,053 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:21,053 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:21,053 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:21,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:21,053 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:21,054 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-26 13:56:21,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:21,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:21,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:21,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:21,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:21,054 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:21,069 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303260ed61c8676cb4a21aa1d9fa47f9111cc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260ed61c8676cb4a21aa1d9fa47f9111cc 2023-03-26 13:56:21,081 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1230, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/efdf28276a904972a3e407b169686bc0 2023-03-26 13:56:21,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:21,116 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:21,134 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:21,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 505 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839041132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:21,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 647 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839041132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:21,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839041134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:21,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839041135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:21,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 507 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839041236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,237 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:21,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 649 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839041236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:21,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839041236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:21,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839041238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:21,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 651 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839041438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:21,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 509 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839041438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:21,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839041440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:21,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839041442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,494 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1230 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/45ef37e13b37403593fe0e9018cef616 2023-03-26 13:56:21,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:21,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46388 deadline: 1679839041730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,733 DEBUG [Thread-4359] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8189 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., hostname=jenkins-hbase4.apache.org,42517,1679838814721, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor35.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 13:56:21,742 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:21,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 653 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839041740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:21,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 511 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839041744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:21,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839041746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:21,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839041746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,903 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1230 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/5cc807e528bf47559b3da507a9a8b5b1 2023-03-26 13:56:21,907 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/efdf28276a904972a3e407b169686bc0 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/efdf28276a904972a3e407b169686bc0 2023-03-26 13:56:21,910 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/efdf28276a904972a3e407b169686bc0, entries=150, sequenceid=1230, filesize=25.4 K 2023-03-26 13:56:21,911 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/45ef37e13b37403593fe0e9018cef616 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/45ef37e13b37403593fe0e9018cef616 2023-03-26 13:56:21,914 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/45ef37e13b37403593fe0e9018cef616, entries=150, sequenceid=1230, filesize=11.9 K 2023-03-26 13:56:21,915 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/5cc807e528bf47559b3da507a9a8b5b1 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5cc807e528bf47559b3da507a9a8b5b1 2023-03-26 13:56:21,918 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5cc807e528bf47559b3da507a9a8b5b1, entries=150, sequenceid=1230, filesize=11.9 K 2023-03-26 13:56:21,918 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for f904470a09c7c6a52bdab844746de048 in 864ms, sequenceid=1230, compaction requested=true 2023-03-26 13:56:21,918 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:21,918 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:21,923 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:21,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:21,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b632f55[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:21,925 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:21,925 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:21,926 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:21,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:21,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:21,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:21,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:21,926 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,926 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:21,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:21,928 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:21,928 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:21,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:21,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:21,930 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:21,930 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:21,930 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:21,930 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:21,930 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:21,934 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:21,934 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:21,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:21,934 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:21,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:21,935 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:21,936 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,938 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,938 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:21,938 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:21,938 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:21,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:21,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:21,938 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:21,938 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:21,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:21,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:21,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:21,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:21,939 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:21,939 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:21,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:21,939 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:21,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:21,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:21,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:22,039 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:22,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:22,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:22,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:22,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:22,042 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:22,042 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:22,042 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:22,042 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:22,043 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,043 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,044 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:22,044 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:22,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,045 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,045 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:22,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:22,046 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:22,046 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:22,046 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:22,046 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:22,046 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:22,047 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-26 13:56:22,047 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:22,047 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:22,047 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:22,047 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:22,047 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:22,047 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:22,062 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326d54f2bbdca384a39bfd1112650f32745 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326d54f2bbdca384a39bfd1112650f32745 2023-03-26 13:56:22,086 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1249, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/fff8943ea46a4b54bdea14bf68a396f2 2023-03-26 13:56:22,097 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1249 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/f9b68eaa74434341a115170365307d8f 2023-03-26 13:56:22,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:22,248 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:22,258 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 516 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839042255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,259 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839042256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 659 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839042258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839042258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 518 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839042359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839042360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 661 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839042361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839042361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,507 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1249 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/fdba608112264d99b20a35c81e4e801b 2023-03-26 13:56:22,511 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/fff8943ea46a4b54bdea14bf68a396f2 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/fff8943ea46a4b54bdea14bf68a396f2 2023-03-26 13:56:22,515 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/fff8943ea46a4b54bdea14bf68a396f2, entries=150, sequenceid=1249, filesize=25.4 K 2023-03-26 13:56:22,515 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/f9b68eaa74434341a115170365307d8f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/f9b68eaa74434341a115170365307d8f 2023-03-26 13:56:22,519 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/f9b68eaa74434341a115170365307d8f, entries=150, sequenceid=1249, filesize=11.9 K 2023-03-26 13:56:22,520 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/fdba608112264d99b20a35c81e4e801b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/fdba608112264d99b20a35c81e4e801b 2023-03-26 13:56:22,523 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/fdba608112264d99b20a35c81e4e801b, entries=150, sequenceid=1249, filesize=11.9 K 2023-03-26 13:56:22,524 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for f904470a09c7c6a52bdab844746de048 in 478ms, sequenceid=1249, compaction requested=true 2023-03-26 13:56:22,524 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:22,524 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:22,524 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:22,524 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:22,524 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,524 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:22,524 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:22,526 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,526 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:22,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:22,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2390d9a3[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:22,528 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:22,528 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,531 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,531 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:22,531 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:22,531 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:22,531 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,531 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:22,532 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,532 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:22,532 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:22,532 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:22,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:22,536 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:22,536 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:22,536 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:22,536 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:22,536 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:22,538 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,538 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:22,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,538 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:22,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:22,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:22,539 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:22,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,541 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:22,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,543 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,543 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,543 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:22,543 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:22,543 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:22,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:22,543 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:22,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:22,544 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:22,543 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:22,544 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:22,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:22,543 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:22,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:22,544 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,544 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:22,563 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-26 13:56:22,563 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:22,563 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:22,563 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:22,563 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:22,563 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:22,563 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:22,579 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303261b47c3c9890640869038377bc47d12ff to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303261b47c3c9890640869038377bc47d12ff 2023-03-26 13:56:22,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839042617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,622 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 525 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839042618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,625 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 667 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839042621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839042622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,644 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:22,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:22,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:22,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:22,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:22,647 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:22,647 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:22,647 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:22,647 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:22,648 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,648 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,650 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:22,650 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:22,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:22,650 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,650 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:22,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:22,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:22,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:22,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:22,651 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:22,651 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:22,651 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:22,651 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:22,651 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:22,651 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:22,651 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:22,652 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:22,652 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:22,652 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:22,652 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:22,652 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,652 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:22,652 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:22,653 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,654 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:22,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:22,654 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,654 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:22,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:22,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:22,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5eeb0d53[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:22,655 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:22,656 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,657 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,657 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,657 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:22,657 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:22,657 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:22,657 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:22,658 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:22,658 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:22,658 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:22,658 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:22,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:22,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:22,661 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:22,661 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:22,661 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:22,661 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:22,661 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:22,663 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,663 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,663 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:22,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:22,664 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,665 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:22,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,678 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:22,678 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:22,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:22,678 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:22,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:22,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:22,679 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:22,679 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:22,678 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,679 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,679 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,679 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,679 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,679 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,679 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:22,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839042723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 527 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839042723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 669 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839042726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839042728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,779 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:22,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:22,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:22,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:22,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:22,782 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:22,782 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:22,782 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:22,782 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:22,783 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,783 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,784 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:22,784 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:22,785 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,785 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:22,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:22,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:22,786 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:22,786 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:22,786 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:22,786 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:22,786 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:22,786 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:22,786 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:22,786 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:22,786 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:22,786 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:22,786 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:22,787 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,787 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:22,787 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:22,788 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,788 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:22,789 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,789 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:22,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:22,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4308a0a2[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:22,790 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:22,790 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,793 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,793 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,793 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,793 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:22,793 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:22,793 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:22,797 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,797 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:22,797 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:22,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:22,797 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:22,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:22,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:22,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:22,800 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:22,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:22,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:22,800 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:22,800 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:22,802 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,802 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:22,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,802 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:22,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:22,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:22,803 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:22,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:22,804 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:22,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,806 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,806 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,806 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,806 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,806 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:22,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:22,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:22,806 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,806 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,806 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:22,806 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,806 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:22,806 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:22,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:22,806 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:22,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:22,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:22,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:22,907 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:22,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:22,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:22,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:22,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:22,910 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:22,910 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:22,910 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:22,910 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:22,911 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,911 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,913 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:22,913 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:22,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:22,913 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,913 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:22,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:22,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:22,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:22,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:22,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:22,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:22,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:22,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:22,915 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:22,915 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:22,915 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:22,915 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:22,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:22,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:22,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:22,915 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:22,917 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,917 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:22,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:22,917 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,917 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:22,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:22,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:22,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71a32611[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:22,919 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:22,919 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,920 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,920 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:22,920 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:22,920 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,920 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:22,920 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:22,922 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,922 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:22,922 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:22,922 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:22,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:22,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:22,924 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:22,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:22,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:22,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:22,924 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:22,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 529 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839042926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,927 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,927 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,927 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:22,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:22,927 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:22,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,928 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,929 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839042926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:22,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:22,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:22,931 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:22,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:22,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:22,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:22,932 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:22,932 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:22,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:22,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:22,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:22,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 671 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839042930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,932 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:22,932 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,932 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,932 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:22,933 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:22,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:22,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:22,934 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:22,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839042932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:22,985 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1263, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/79107eb01d3f4804b0a08b1ab8b636f4 2023-03-26 13:56:22,994 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1263 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/cbfd0f4c689e4f2889fcb43e883691bb 2023-03-26 13:56:23,003 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=1263 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/49a708e6928d43bf8a2f1a8f58abee24 2023-03-26 13:56:23,007 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/79107eb01d3f4804b0a08b1ab8b636f4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/79107eb01d3f4804b0a08b1ab8b636f4 2023-03-26 13:56:23,010 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/79107eb01d3f4804b0a08b1ab8b636f4, entries=150, sequenceid=1263, filesize=25.4 K 2023-03-26 13:56:23,011 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/cbfd0f4c689e4f2889fcb43e883691bb as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/cbfd0f4c689e4f2889fcb43e883691bb 2023-03-26 13:56:23,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/cbfd0f4c689e4f2889fcb43e883691bb, entries=150, sequenceid=1263, filesize=11.9 K 2023-03-26 13:56:23,015 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/49a708e6928d43bf8a2f1a8f58abee24 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/49a708e6928d43bf8a2f1a8f58abee24 2023-03-26 13:56:23,019 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/49a708e6928d43bf8a2f1a8f58abee24, entries=150, sequenceid=1263, filesize=11.9 K 2023-03-26 13:56:23,019 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for f904470a09c7c6a52bdab844746de048 in 456ms, sequenceid=1263, compaction requested=true 2023-03-26 13:56:23,019 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:23,019 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:23,019 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:23,019 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 13:56:23,019 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 13:56:23,019 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:23,021 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 157060 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 13:56:23,021 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/A is initiating minor compaction (all files) 2023-03-26 13:56:23,021 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/A in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,021 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73900 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 13:56:23,021 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/54da4f30389e44f8a068b8b1d23a8aed, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/04a2b7185a5c4727a42c62e84e1916ad, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/202cbfcaebca444cb921ad780065d964, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/efdf28276a904972a3e407b169686bc0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/fff8943ea46a4b54bdea14bf68a396f2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/79107eb01d3f4804b0a08b1ab8b636f4] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=153.4 K 2023-03-26 13:56:23,021 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/B is initiating minor compaction (all files) 2023-03-26 13:56:23,022 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/B in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,022 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/301d14ff980045e78380989fb48c898b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1d4e38503fc74fdf89b8ff9a30f61d93, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/074027c947aa4867b7a6cb93103612ef, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/45ef37e13b37403593fe0e9018cef616, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/f9b68eaa74434341a115170365307d8f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/cbfd0f4c689e4f2889fcb43e883691bb] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=72.2 K 2023-03-26 13:56:23,022 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 54da4f30389e44f8a068b8b1d23a8aed, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=1171, earliestPutTs=1679838978334 2023-03-26 13:56:23,022 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 301d14ff980045e78380989fb48c898b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=1171, earliestPutTs=1679838978334 2023-03-26 13:56:23,022 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 04a2b7185a5c4727a42c62e84e1916ad, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1182, earliestPutTs=1679838979462 2023-03-26 13:56:23,022 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 1d4e38503fc74fdf89b8ff9a30f61d93, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1182, earliestPutTs=1679838979462 2023-03-26 13:56:23,023 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 202cbfcaebca444cb921ad780065d964, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1217, earliestPutTs=1679838979615 2023-03-26 13:56:23,023 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 074027c947aa4867b7a6cb93103612ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1217, earliestPutTs=1679838979615 2023-03-26 13:56:23,023 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting efdf28276a904972a3e407b169686bc0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1230, earliestPutTs=1679838980774 2023-03-26 13:56:23,024 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 45ef37e13b37403593fe0e9018cef616, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1230, earliestPutTs=1679838980774 2023-03-26 13:56:23,024 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting f9b68eaa74434341a115170365307d8f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1249, earliestPutTs=1679838981129 2023-03-26 13:56:23,024 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting fff8943ea46a4b54bdea14bf68a396f2, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1249, earliestPutTs=1679838981129 2023-03-26 13:56:23,024 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting cbfd0f4c689e4f2889fcb43e883691bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1263, earliestPutTs=1679838982254 2023-03-26 13:56:23,024 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 79107eb01d3f4804b0a08b1ab8b636f4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1263, earliestPutTs=1679838982254 2023-03-26 13:56:23,032 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:23,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:23,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:23,035 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#B#compaction#670 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:23,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:23,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:23,036 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:23,036 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:23,036 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:23,036 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:23,036 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,036 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,037 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#A#compaction#669 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:23,037 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:23,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:23,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:23,037 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,038 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,038 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:23,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:23,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:23,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:23,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:23,044 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:23,050 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:23,050 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:23,050 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,050 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:23,050 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-26 13:56:23,050 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:23,050 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:23,050 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:23,051 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:23,051 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:23,051 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:23,058 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/9b627a0acd9b485295d73d8d22ef5654 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/9b627a0acd9b485295d73d8d22ef5654 2023-03-26 13:56:23,062 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f904470a09c7c6a52bdab844746de048/B of f904470a09c7c6a52bdab844746de048 into 9b627a0acd9b485295d73d8d22ef5654(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:23,062 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:23,062 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/B, priority=10, startTime=1679838983019; duration=0sec 2023-03-26 13:56:23,062 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:23,062 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 13:56:23,065 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73900 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 13:56:23,065 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/C is initiating minor compaction (all files) 2023-03-26 13:56:23,066 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/C in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,066 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/1308f8b1e31a473abdeb636f10e5c77c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/a7fb8a70234a4cd08f7c84179546b1ea, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/246e7b75270c45878fda2544e269e9c5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5cc807e528bf47559b3da507a9a8b5b1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/fdba608112264d99b20a35c81e4e801b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/49a708e6928d43bf8a2f1a8f58abee24] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=72.2 K 2023-03-26 13:56:23,070 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 1308f8b1e31a473abdeb636f10e5c77c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=1171, earliestPutTs=1679838978334 2023-03-26 13:56:23,081 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting a7fb8a70234a4cd08f7c84179546b1ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1182, earliestPutTs=1679838979462 2023-03-26 13:56:23,081 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 246e7b75270c45878fda2544e269e9c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1217, earliestPutTs=1679838979615 2023-03-26 13:56:23,082 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 5cc807e528bf47559b3da507a9a8b5b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1230, earliestPutTs=1679838980774 2023-03-26 13:56:23,082 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting fdba608112264d99b20a35c81e4e801b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1249, earliestPutTs=1679838981129 2023-03-26 13:56:23,082 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 49a708e6928d43bf8a2f1a8f58abee24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1263, earliestPutTs=1679838982254 2023-03-26 13:56:23,090 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#C#compaction#672 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:23,093 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326f737d4a1ec834650979c48940809d140 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f737d4a1ec834650979c48940809d140 2023-03-26 13:56:23,099 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/315cb44a48a247b8a05e0241650c373b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/315cb44a48a247b8a05e0241650c373b 2023-03-26 13:56:23,102 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f904470a09c7c6a52bdab844746de048/C of f904470a09c7c6a52bdab844746de048 into 315cb44a48a247b8a05e0241650c373b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:23,102 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:23,102 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/C, priority=10, startTime=1679838983019; duration=0sec 2023-03-26 13:56:23,102 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:23,105 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1282, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/a5d64006bdbf422b818b10445df8cf04 2023-03-26 13:56:23,117 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1282 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/32d52bb015804e28b0de93c0333942c4 2023-03-26 13:56:23,127 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=1282 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/35c7910ffb694d0c954c8716a7da2912 2023-03-26 13:56:23,131 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/a5d64006bdbf422b818b10445df8cf04 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/a5d64006bdbf422b818b10445df8cf04 2023-03-26 13:56:23,134 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/a5d64006bdbf422b818b10445df8cf04, entries=150, sequenceid=1282, filesize=25.4 K 2023-03-26 13:56:23,135 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/32d52bb015804e28b0de93c0333942c4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/32d52bb015804e28b0de93c0333942c4 2023-03-26 13:56:23,138 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/32d52bb015804e28b0de93c0333942c4, entries=150, sequenceid=1282, filesize=11.9 K 2023-03-26 13:56:23,139 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/35c7910ffb694d0c954c8716a7da2912 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/35c7910ffb694d0c954c8716a7da2912 2023-03-26 13:56:23,145 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/35c7910ffb694d0c954c8716a7da2912, entries=150, sequenceid=1282, filesize=11.9 K 2023-03-26 13:56:23,146 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for f904470a09c7c6a52bdab844746de048 in 96ms, sequenceid=1282, compaction requested=false 2023-03-26 13:56:23,146 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:23,146 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,146 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:23,146 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:23,146 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,146 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:23,146 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,148 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,148 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:23,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:23,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:23,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@545bb447[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:23,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:23,150 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,151 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,152 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:23,152 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,152 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:23,152 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:23,152 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:23,153 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:23,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:23,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:23,153 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:23,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:23,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:23,156 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:23,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:23,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:23,156 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:23,156 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:23,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,158 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:23,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:23,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:23,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,159 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:23,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:23,160 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:23,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:23,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,161 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,164 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,164 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:23,164 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:23,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:23,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:23,164 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:23,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:23,164 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:23,164 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:23,164 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,164 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,164 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:23,164 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:23,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:23,164 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,165 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:23,252 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:56:23,253 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:23,253 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:23,253 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:23,253 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:23,253 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:23,253 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:23,265 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:23,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:23,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:23,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:23,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:23,269 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:23,269 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:23,269 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:23,270 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:23,270 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,270 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,272 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032679246fe630bc4c0ab96051a2c9ab2ffd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032679246fe630bc4c0ab96051a2c9ab2ffd 2023-03-26 13:56:23,272 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,273 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:23,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:23,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,273 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,273 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:23,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:23,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:23,274 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:23,274 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:23,274 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:23,274 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:23,274 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:23,274 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:23,274 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,274 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:23,274 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:23,275 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:23,275 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:23,275 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:23,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:23,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839043272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679838983268, max timestamp=1679838983275Num uniques 859; 2023-03-26 13:56:23,277 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:23,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838983268, max timestamp=1679838983275Num uniques 859; 2023-03-26 13:56:23,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:23,278 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,278 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,278 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:23,277 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838983268, max timestamp=1679838983275Num uniques 859; 2023-03-26 13:56:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:23,278 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:23,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:23,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52dcba0a[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:23,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:23,280 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,282 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:23,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:23,282 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:23,282 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1295, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/cd5a33cda9d44e8495937913b288cc22 2023-03-26 13:56:23,283 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,283 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:23,284 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:23,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:23,284 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:23,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:23,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:23,290 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:23,290 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:23,290 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:23,290 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:23,290 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:23,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:23,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,295 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:23,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:23,296 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:23,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,297 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,299 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,299 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:23,300 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:23,300 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:23,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:23,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:23,299 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,300 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,300 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,300 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,300 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,300 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:23,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:23,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:23,300 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:23,300 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:23,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:23,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:23,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:23,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 543 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839043303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:23,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 685 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839043304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:23,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839043304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:23,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839043382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,400 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:23,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:23,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:23,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:23,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:23,404 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:23,404 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:23,404 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:23,404 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:23,404 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,404 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,406 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:23,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:23,406 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,406 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,406 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:23,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:23,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:23,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:23,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:23,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:23,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:23,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:23,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:23,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:23,409 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,409 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:23,409 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:23,409 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,409 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:23,410 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:23,410 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,410 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:23,410 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:23,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:23,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 545 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839043409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:23,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 687 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839043410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:23,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839043410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,412 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,412 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:23,413 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,413 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:23,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:23,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54571d43[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:23,414 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:23,414 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,415 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,415 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:23,415 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:23,416 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:23,416 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,416 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:23,417 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,417 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:23,417 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:23,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:23,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:23,417 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:23,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:23,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:23,419 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:23,419 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:23,419 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:23,420 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:23,420 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:23,421 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,421 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:23,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:23,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,421 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:23,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:23,422 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,423 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,426 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,426 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,426 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:23,426 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:23,426 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:23,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:23,427 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,427 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:23,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:23,427 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,427 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:23,427 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,427 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:23,427 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:23,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:23,489 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/51218eba893043d7b7b8a96a024f48f9 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/51218eba893043d7b7b8a96a024f48f9 2023-03-26 13:56:23,493 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f904470a09c7c6a52bdab844746de048/A of f904470a09c7c6a52bdab844746de048 into 51218eba893043d7b7b8a96a024f48f9(size=26.4 K), total size for store is 51.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:23,493 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:23,493 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/A, priority=10, startTime=1679838983019; duration=0sec 2023-03-26 13:56:23,494 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:23,527 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:23,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:23,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:23,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:23,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:23,530 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:23,530 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:23,530 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:23,530 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:23,531 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,531 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,532 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:23,532 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:23,532 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,532 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:23,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,533 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:23,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,533 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:23,533 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:23,533 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:23,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:23,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:23,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:23,533 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:23,534 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:23,534 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,534 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:23,534 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:23,534 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,534 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:23,534 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:23,534 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,534 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:23,534 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:23,535 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,535 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:23,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:23,536 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,536 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:23,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:23,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cac2a76[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:23,537 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:23,538 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,539 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,539 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:23,539 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:23,539 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:23,539 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,539 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:23,542 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:23,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:23,542 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:23,542 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:23,542 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:23,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:23,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:23,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:23,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:23,545 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:23,545 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:23,545 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:23,545 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:23,545 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:23,545 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:23,548 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,548 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,548 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:23,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:23,549 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:23,550 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:23,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,553 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:23,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:23,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:23,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:23,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:23,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:23,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,553 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:23,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:23,553 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:23,553 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:23,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:23,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:23,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:23,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839043586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:23,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 547 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839043613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:23,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 689 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839043613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:23,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839043613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,654 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:23,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:23,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:23,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:23,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:23,657 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:23,657 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:23,657 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:23,657 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:23,657 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,657 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,659 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:23,659 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:23,659 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,659 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:23,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:23,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,660 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:23,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:23,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:23,661 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:23,661 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:23,661 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:23,661 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:23,661 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,661 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:23,661 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:23,661 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:23,661 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,661 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:23,661 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:23,661 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,661 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:23,662 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:23,664 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,664 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:23,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:23,664 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,664 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:23,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:23,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:23,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:23,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:23,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@355c5828[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:23,666 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:23,666 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,668 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,668 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:23,668 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:23,668 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:23,668 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,668 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:23,669 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:23,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:23,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:23,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:23,669 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:23,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:23,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:23,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:23,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:23,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:23,672 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:23,672 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:23,672 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:23,672 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:23,672 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:23,673 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,673 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:23,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,673 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:23,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:23,674 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:23,676 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,681 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,681 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,681 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,681 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:23,681 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,682 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,682 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,682 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:23,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:23,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:23,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:23,682 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:23,682 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:23,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:23,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:23,682 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,682 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:23,700 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1295 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/dfaa425c198e4de5a60fe0cf1afeb38f 2023-03-26 13:56:23,708 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1295 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/04ce652266cc44248c9f965814de4abf 2023-03-26 13:56:23,712 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/cd5a33cda9d44e8495937913b288cc22 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/cd5a33cda9d44e8495937913b288cc22 2023-03-26 13:56:23,715 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/cd5a33cda9d44e8495937913b288cc22, entries=200, sequenceid=1295, filesize=32.3 K 2023-03-26 13:56:23,716 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/dfaa425c198e4de5a60fe0cf1afeb38f as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/dfaa425c198e4de5a60fe0cf1afeb38f 2023-03-26 13:56:23,719 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/dfaa425c198e4de5a60fe0cf1afeb38f, entries=150, sequenceid=1295, filesize=11.9 K 2023-03-26 13:56:23,719 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/04ce652266cc44248c9f965814de4abf as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/04ce652266cc44248c9f965814de4abf 2023-03-26 13:56:23,725 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/04ce652266cc44248c9f965814de4abf, entries=150, sequenceid=1295, filesize=11.9 K 2023-03-26 13:56:23,726 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=129.71 KB/132820 for f904470a09c7c6a52bdab844746de048 in 474ms, sequenceid=1295, compaction requested=true 2023-03-26 13:56:23,726 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:23,726 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:23,726 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:56:23,726 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:56:23,726 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:23,726 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:23,727 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86166 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:56:23,727 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/A is initiating minor compaction (all files) 2023-03-26 13:56:23,727 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/A in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,727 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/51218eba893043d7b7b8a96a024f48f9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/a5d64006bdbf422b818b10445df8cf04, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/cd5a33cda9d44e8495937913b288cc22] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=84.1 K 2023-03-26 13:56:23,728 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:56:23,728 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/B is initiating minor compaction (all files) 2023-03-26 13:56:23,728 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/B in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,728 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/9b627a0acd9b485295d73d8d22ef5654, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/32d52bb015804e28b0de93c0333942c4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/dfaa425c198e4de5a60fe0cf1afeb38f] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=36.7 K 2023-03-26 13:56:23,728 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 51218eba893043d7b7b8a96a024f48f9, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=1263, earliestPutTs=1679838982254 2023-03-26 13:56:23,728 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 9b627a0acd9b485295d73d8d22ef5654, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1263, earliestPutTs=1679838982254 2023-03-26 13:56:23,729 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting a5d64006bdbf422b818b10445df8cf04, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1282, earliestPutTs=1679838982616 2023-03-26 13:56:23,729 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 32d52bb015804e28b0de93c0333942c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1282, earliestPutTs=1679838982616 2023-03-26 13:56:23,729 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting cd5a33cda9d44e8495937913b288cc22, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1295, earliestPutTs=1679838983241 2023-03-26 13:56:23,729 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting dfaa425c198e4de5a60fe0cf1afeb38f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1295, earliestPutTs=1679838983241 2023-03-26 13:56:23,735 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#B#compaction#678 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:23,739 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#A#compaction#679 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:23,748 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/99d2ff895b1d4999a97c223f0e1c37b3 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/99d2ff895b1d4999a97c223f0e1c37b3 2023-03-26 13:56:23,754 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f904470a09c7c6a52bdab844746de048/B of f904470a09c7c6a52bdab844746de048 into 99d2ff895b1d4999a97c223f0e1c37b3(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:23,754 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:23,754 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/B, priority=13, startTime=1679838983726; duration=0sec 2023-03-26 13:56:23,754 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:23,754 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:56:23,755 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:56:23,755 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/C is initiating minor compaction (all files) 2023-03-26 13:56:23,755 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/C in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,755 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/315cb44a48a247b8a05e0241650c373b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/35c7910ffb694d0c954c8716a7da2912, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/04ce652266cc44248c9f965814de4abf] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=36.7 K 2023-03-26 13:56:23,756 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 315cb44a48a247b8a05e0241650c373b, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=1263, earliestPutTs=1679838982254 2023-03-26 13:56:23,756 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 35c7910ffb694d0c954c8716a7da2912, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1282, earliestPutTs=1679838982616 2023-03-26 13:56:23,756 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 04ce652266cc44248c9f965814de4abf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1295, earliestPutTs=1679838983241 2023-03-26 13:56:23,761 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#C#compaction#680 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:23,778 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/e2476138db9b4111916aea48abf77b23 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/e2476138db9b4111916aea48abf77b23 2023-03-26 13:56:23,782 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f904470a09c7c6a52bdab844746de048/C of f904470a09c7c6a52bdab844746de048 into e2476138db9b4111916aea48abf77b23(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:23,782 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:23,782 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/C, priority=13, startTime=1679838983726; duration=0sec 2023-03-26 13:56:23,782 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:23,782 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:23,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:23,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:23,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:23,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:23,786 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:23,786 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:23,786 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:23,786 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:23,786 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,786 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,788 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,788 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:23,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:23,788 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,788 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:23,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:23,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:23,790 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:23,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:23,790 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:23,790 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:23,790 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:23,790 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:23,790 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,790 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:23,790 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:23,790 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=129.71 KB heapSize=370.14 KB 2023-03-26 13:56:23,791 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:23,791 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:23,791 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:23,791 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:23,791 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:23,791 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:23,809 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032698c457567f2c4d75b01b4b4e67231520 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032698c457567f2c4d75b01b4b4e67231520 2023-03-26 13:56:23,818 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1334, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/add1320c10bb4e7ea8aa100798464170 2023-03-26 13:56:23,837 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1334 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/3901a9ce446b452298007bb76396c3f8 2023-03-26 13:56:23,853 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1334 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/96cac0dbb3e3407fae9395dd56f62ceb 2023-03-26 13:56:23,857 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/add1320c10bb4e7ea8aa100798464170 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/add1320c10bb4e7ea8aa100798464170 2023-03-26 13:56:23,861 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/add1320c10bb4e7ea8aa100798464170, entries=150, sequenceid=1334, filesize=25.4 K 2023-03-26 13:56:23,861 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/3901a9ce446b452298007bb76396c3f8 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/3901a9ce446b452298007bb76396c3f8 2023-03-26 13:56:23,871 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/3901a9ce446b452298007bb76396c3f8, entries=150, sequenceid=1334, filesize=11.9 K 2023-03-26 13:56:23,871 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/96cac0dbb3e3407fae9395dd56f62ceb as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/96cac0dbb3e3407fae9395dd56f62ceb 2023-03-26 13:56:23,883 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/96cac0dbb3e3407fae9395dd56f62ceb, entries=150, sequenceid=1334, filesize=11.9 K 2023-03-26 13:56:23,883 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~129.71 KB/132820, heapSize ~370.09 KB/378976, currentSize=0 B/0 for f904470a09c7c6a52bdab844746de048 in 93ms, sequenceid=1334, compaction requested=false 2023-03-26 13:56:23,884 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:23,884 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,884 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:23,884 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:23,884 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:23,888 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:23,890 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,890 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:23,890 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,890 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:23,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:23,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@267b863[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:23,892 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:23,892 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,899 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,899 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:23,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:23,900 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,900 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:23,900 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:23,901 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,901 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:23,901 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:23,901 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:23,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:23,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:23,904 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:23,904 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:23,904 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:23,904 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:23,904 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:23,905 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,905 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:23,905 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:23,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:23,906 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:23,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:23,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,907 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:23,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:23,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,909 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:23,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:23,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:23,910 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:23,909 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,910 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,910 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:23,910 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,910 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,910 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:23,910 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:23,910 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:23,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:23,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:23,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:23,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:23,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:23,910 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:23,910 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:23,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:23,935 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:56:23,935 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:23,935 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:23,935 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:23,935 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:23,935 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:23,935 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:23,957 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326e6fc74309e8841e296a48cf29830699f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e6fc74309e8841e296a48cf29830699f 2023-03-26 13:56:23,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:23,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 697 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839043953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:23,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 555 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839043954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:23,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838983948, max timestamp=1679838983957Num uniques 859; 2023-03-26 13:56:23,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838983948, max timestamp=1679838983957Num uniques 859; 2023-03-26 13:56:23,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:23,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:23,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838983948, max timestamp=1679838983957Num uniques 859; 2023-03-26 13:56:23,959 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:23,965 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1344, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/9ddec078aa52461880fed76f6d87e4d6 2023-03-26 13:56:23,973 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1344 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/ae3f03375c8b42028cb9fc3d25caaccf 2023-03-26 13:56:23,981 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=1344 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/d9071412486c408eb5fc777fadfa8d11 2023-03-26 13:56:23,984 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/9ddec078aa52461880fed76f6d87e4d6 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/9ddec078aa52461880fed76f6d87e4d6 2023-03-26 13:56:23,989 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/9ddec078aa52461880fed76f6d87e4d6, entries=350, sequenceid=1344, filesize=52.9 K 2023-03-26 13:56:23,989 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/ae3f03375c8b42028cb9fc3d25caaccf as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/ae3f03375c8b42028cb9fc3d25caaccf 2023-03-26 13:56:23,992 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/ae3f03375c8b42028cb9fc3d25caaccf, entries=150, sequenceid=1344, filesize=11.9 K 2023-03-26 13:56:23,993 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/d9071412486c408eb5fc777fadfa8d11 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/d9071412486c408eb5fc777fadfa8d11 2023-03-26 13:56:23,996 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/d9071412486c408eb5fc777fadfa8d11, entries=150, sequenceid=1344, filesize=11.9 K 2023-03-26 13:56:23,997 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f904470a09c7c6a52bdab844746de048 in 63ms, sequenceid=1344, compaction requested=true 2023-03-26 13:56:23,997 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:23,997 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:23,997 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:23,997 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-26 13:56:23,997 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-03-26 13:56:23,997 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 13:56:23,997 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 13:56:23,998 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. because compaction request was cancelled 2023-03-26 13:56:23,998 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:56:23,999 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:56:23,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:23,999 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=147.60 KB heapSize=425.53 KB 2023-03-26 13:56:23,999 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/B is initiating minor compaction (all files) 2023-03-26 13:56:23,999 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/B in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:23,999 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/99d2ff895b1d4999a97c223f0e1c37b3, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/3901a9ce446b452298007bb76396c3f8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/ae3f03375c8b42028cb9fc3d25caaccf] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=36.8 K 2023-03-26 13:56:23,999 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:24,000 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-26 13:56:24,000 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 99d2ff895b1d4999a97c223f0e1c37b3, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=1295, earliestPutTs=1679838983241 2023-03-26 13:56:24,000 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:24,000 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-26 13:56:24,000 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:24,000 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-26 13:56:24,000 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 3901a9ce446b452298007bb76396c3f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1334, earliestPutTs=1679838983297 2023-03-26 13:56:24,001 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting ae3f03375c8b42028cb9fc3d25caaccf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1344, earliestPutTs=1679838983931 2023-03-26 13:56:24,010 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:24,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:24,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839044009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:24,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:24,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:24,014 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:24,014 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:24,014 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:24,014 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:24,014 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,014 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839044011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,017 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:24,017 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:24,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:24,017 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,017 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:24,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:24,029 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#B#compaction#688 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:24,035 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:24,035 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:24,035 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:24,035 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:24,035 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:24,036 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:24,036 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:24,036 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:24,037 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:24,037 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:24,037 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:24,037 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:24,037 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:24,037 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,037 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:24,037 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:24,038 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,038 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:24,039 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:24,039 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:24,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:24,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:24,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e360069[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:24,041 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:24,042 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,043 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:24,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:24,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:24,045 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,045 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:24,047 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,047 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:24,047 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:24,047 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:24,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:24,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:24,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,048 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032638101cf7bd514654977e23dc5076c3ee to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032638101cf7bd514654977e23dc5076c3ee 2023-03-26 13:56:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:24,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:24,050 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:24,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:24,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:24,050 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:24,050 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:24,053 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,053 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:24,053 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:24,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:24,053 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:24,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:24,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:24,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:24,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,056 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,059 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:24,059 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:24,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:24,059 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,059 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,059 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,059 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,059 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,059 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,059 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,059 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:24,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:24,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:24,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:24,060 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:24,060 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:24,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:24,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:24,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:24,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 557 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839044060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 699 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839044060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,074 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1383, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/917d4ebc10d54df8b9d6c409657380f1 2023-03-26 13:56:24,093 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1383 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/113887428aca49c6b9a69b8cf85b2dc9 2023-03-26 13:56:24,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839044113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839044117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,160 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:24,160 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/71618c6d257f41519a6cef2e38f719b1 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/71618c6d257f41519a6cef2e38f719b1 2023-03-26 13:56:24,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:24,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:24,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:24,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:24,163 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:24,163 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:24,164 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:24,164 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:24,164 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,164 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,165 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f904470a09c7c6a52bdab844746de048/A of f904470a09c7c6a52bdab844746de048 into 71618c6d257f41519a6cef2e38f719b1(size=26.5 K), total size for store is 104.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:24,165 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:24,165 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/A, priority=13, startTime=1679838983726; duration=0sec 2023-03-26 13:56:24,165 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:24,165 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 13:56:24,166 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:24,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:24,166 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:24,166 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,166 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:24,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,166 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 13:56:24,166 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/C is initiating minor compaction (all files) 2023-03-26 13:56:24,166 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/C in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:24,167 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/e2476138db9b4111916aea48abf77b23, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/96cac0dbb3e3407fae9395dd56f62ceb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/d9071412486c408eb5fc777fadfa8d11] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=36.8 K 2023-03-26 13:56:24,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,167 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting e2476138db9b4111916aea48abf77b23, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=1295, earliestPutTs=1679838983241 2023-03-26 13:56:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:24,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,167 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 96cac0dbb3e3407fae9395dd56f62ceb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1334, earliestPutTs=1679838983297 2023-03-26 13:56:24,167 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:24,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:24,168 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting d9071412486c408eb5fc777fadfa8d11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1344, earliestPutTs=1679838983931 2023-03-26 13:56:24,168 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:24,168 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:24,168 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:24,168 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:24,172 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:24,172 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:24,172 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:24,172 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:24,172 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:24,172 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:24,172 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:24,172 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:24,172 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,172 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:24,172 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:24,173 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#C#compaction#691 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:24,174 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,174 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:24,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:24,174 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,174 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:24,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:24,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:24,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:24,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:24,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33947843[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:24,176 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:24,176 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,178 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,178 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:24,178 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:24,178 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:24,178 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,178 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:24,179 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,179 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:24,179 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:24,179 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:24,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:24,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:24,181 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:24,181 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:24,181 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:24,181 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:24,182 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:24,183 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,183 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,183 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:24,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:24,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:24,183 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,185 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,187 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,187 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,187 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,187 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:24,187 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:24,187 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:24,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:24,187 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,187 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,187 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:24,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:24,187 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:24,187 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:24,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:24,188 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:24,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:24,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:24,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 559 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839044265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 701 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839044266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,288 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:24,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:24,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:24,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:24,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:24,291 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:24,291 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:24,291 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:24,291 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:24,291 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,291 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,297 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,297 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:24,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:24,298 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,298 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:24,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:24,299 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:24,299 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:24,299 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:24,299 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:24,299 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:24,299 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:24,299 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:24,299 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:24,299 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:24,300 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:24,300 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:24,300 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:24,300 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,300 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:24,300 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:24,301 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:24,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:24,301 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:24,302 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,302 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:24,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:24,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64deeb01[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:24,303 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:24,303 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,304 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:24,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:24,304 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:24,306 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,306 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:24,306 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:24,306 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:24,306 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:24,306 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:24,306 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:24,306 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:24,306 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:24,306 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:24,306 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:24,309 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,309 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:24,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,309 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:24,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:24,309 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:24,311 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,313 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,313 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:24,313 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,313 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,313 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,313 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:24,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:24,313 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:24,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:24,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:24,313 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:24,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:24,314 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:24,314 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:24,314 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,314 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839044318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839044321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,414 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:24,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:24,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:24,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:24,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:24,417 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:24,417 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:24,418 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:24,418 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:24,418 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,418 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,419 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:24,419 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:24,420 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,420 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:24,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:24,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,421 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:24,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:24,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:24,421 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:24,421 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:24,421 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:24,421 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:24,421 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:24,421 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:24,421 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:24,421 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:24,421 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:24,422 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:24,422 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:24,422 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,422 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:24,422 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:24,423 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,423 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:24,424 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,424 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:24,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:24,425 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:24,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b45e359[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:24,425 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,427 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,427 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:24,427 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:24,427 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:24,427 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,427 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:24,429 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:24,429 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:24,429 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,429 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:24,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:24,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:24,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:24,431 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:24,431 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:24,431 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:24,431 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:24,431 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:24,433 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,433 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,433 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:24,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:24,434 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:24,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,435 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:24,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:24,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:24,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:24,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:24,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:24,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:24,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:24,438 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:24,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,438 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:24,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:24,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:24,450 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/e34190cebba8417196ca6c31bd6396ef as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/e34190cebba8417196ca6c31bd6396ef 2023-03-26 13:56:24,454 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f904470a09c7c6a52bdab844746de048/B of f904470a09c7c6a52bdab844746de048 into e34190cebba8417196ca6c31bd6396ef(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:24,454 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:24,454 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/B, priority=13, startTime=1679838983997; duration=0sec 2023-03-26 13:56:24,454 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:24,514 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=1383 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/4d303961e5eb43afa951f75f2f00d00a 2023-03-26 13:56:24,518 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/917d4ebc10d54df8b9d6c409657380f1 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/917d4ebc10d54df8b9d6c409657380f1 2023-03-26 13:56:24,521 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/917d4ebc10d54df8b9d6c409657380f1, entries=250, sequenceid=1383, filesize=39.2 K 2023-03-26 13:56:24,521 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/113887428aca49c6b9a69b8cf85b2dc9 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/113887428aca49c6b9a69b8cf85b2dc9 2023-03-26 13:56:24,525 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/113887428aca49c6b9a69b8cf85b2dc9, entries=150, sequenceid=1383, filesize=11.9 K 2023-03-26 13:56:24,525 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/4d303961e5eb43afa951f75f2f00d00a as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/4d303961e5eb43afa951f75f2f00d00a 2023-03-26 13:56:24,529 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/4d303961e5eb43afa951f75f2f00d00a, entries=150, sequenceid=1383, filesize=11.9 K 2023-03-26 13:56:24,529 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~425.48 KB/435696, currentSize=33.54 KB/34350 for f904470a09c7c6a52bdab844746de048 in 530ms, sequenceid=1383, compaction requested=true 2023-03-26 13:56:24,530 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:24,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:24,530 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:56:24,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:24,530 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-26 13:56:24,531 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 147526 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:56:24,531 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/A is initiating minor compaction (all files) 2023-03-26 13:56:24,531 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/A in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:24,531 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/71618c6d257f41519a6cef2e38f719b1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/add1320c10bb4e7ea8aa100798464170, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/9ddec078aa52461880fed76f6d87e4d6, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/917d4ebc10d54df8b9d6c409657380f1] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=144.1 K 2023-03-26 13:56:24,532 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 71618c6d257f41519a6cef2e38f719b1, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=1295, earliestPutTs=1679838983241 2023-03-26 13:56:24,532 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting add1320c10bb4e7ea8aa100798464170, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1334, earliestPutTs=1679838983297 2023-03-26 13:56:24,532 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 9ddec078aa52461880fed76f6d87e4d6, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=1344, earliestPutTs=1679838983911 2023-03-26 13:56:24,533 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 917d4ebc10d54df8b9d6c409657380f1, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=1383, earliestPutTs=1679838983982 2023-03-26 13:56:24,538 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:24,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:24,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:24,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:24,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:24,542 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:24,542 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:24,542 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:24,542 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:24,544 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,544 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,544 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#A#compaction#692 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:24,546 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:24,546 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:24,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:24,546 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,546 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:24,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:24,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:24,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:24,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:24,548 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:24,548 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:24,548 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:24,548 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-26 13:56:24,549 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:24,549 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:24,549 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:24,549 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:24,549 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:24,549 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:24,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:24,572 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:24,581 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/4377046344354ee493e82e1d98f7d644 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/4377046344354ee493e82e1d98f7d644 2023-03-26 13:56:24,585 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f904470a09c7c6a52bdab844746de048/C of f904470a09c7c6a52bdab844746de048 into 4377046344354ee493e82e1d98f7d644(size=13.1 K), total size for store is 25.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:24,585 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:24,585 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/C, priority=13, startTime=1679838983997; duration=0sec 2023-03-26 13:56:24,585 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-26 13:56:24,585 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-26 13:56:24,586 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 13:56:24,586 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 13:56:24,586 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. because compaction request was cancelled 2023-03-26 13:56:24,586 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-26 13:56:24,586 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 13:56:24,586 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 13:56:24,586 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. because compaction request was cancelled 2023-03-26 13:56:24,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838984603, max timestamp=1679838984610Num uniques 859; 2023-03-26 13:56:24,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:24,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679838984603, max timestamp=1679838984610Num uniques 859; 2023-03-26 13:56:24,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 13:56:24,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 716 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839044629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,633 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 576 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839044630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839044631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839044632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,736 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 718 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839044734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 578 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839044734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839044736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839044736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 720 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839044938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839044938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 580 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839044939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:24,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839044941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:24,965 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/fcb88183ed6a41a3aeee9e580a06f3fb as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/fcb88183ed6a41a3aeee9e580a06f3fb 2023-03-26 13:56:24,969 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f904470a09c7c6a52bdab844746de048/A of f904470a09c7c6a52bdab844746de048 into fcb88183ed6a41a3aeee9e580a06f3fb(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:24,969 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:24,969 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/A, priority=12, startTime=1679838984530; duration=0sec 2023-03-26 13:56:24,969 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:24,970 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326e7fbefe4d9ab4fe087df153f78bb3424 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e7fbefe4d9ab4fe087df153f78bb3424 2023-03-26 13:56:25,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:25,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 722 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839045242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:25,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 582 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839045242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:25,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839045243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:25,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839045246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,376 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1393, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/872870fdf8c6441b8e1158b77c9e157c 2023-03-26 13:56:25,750 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:25,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 584 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839045748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:25,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839045748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:25,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 724 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839045749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:25,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839045751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,785 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1393 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/38cdcf56c8d04f57919aa55acf7777a4 2023-03-26 13:56:25,793 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1393 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/37f40c8fa6734189818d5a94ec97ae01 2023-03-26 13:56:25,797 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/872870fdf8c6441b8e1158b77c9e157c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/872870fdf8c6441b8e1158b77c9e157c 2023-03-26 13:56:25,802 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/872870fdf8c6441b8e1158b77c9e157c, entries=100, sequenceid=1393, filesize=18.6 K 2023-03-26 13:56:25,802 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/38cdcf56c8d04f57919aa55acf7777a4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/38cdcf56c8d04f57919aa55acf7777a4 2023-03-26 13:56:25,806 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/38cdcf56c8d04f57919aa55acf7777a4, entries=100, sequenceid=1393, filesize=9.5 K 2023-03-26 13:56:25,806 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/37f40c8fa6734189818d5a94ec97ae01 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/37f40c8fa6734189818d5a94ec97ae01 2023-03-26 13:56:25,809 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/37f40c8fa6734189818d5a94ec97ae01, entries=100, sequenceid=1393, filesize=9.5 K 2023-03-26 13:56:25,810 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=145.36 KB/148850 for f904470a09c7c6a52bdab844746de048 in 1262ms, sequenceid=1393, compaction requested=true 2023-03-26 13:56:25,810 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:25,810 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:25,810 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:25,810 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:25,810 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,810 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:25,810 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:25,812 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,812 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:25,812 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:25,812 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:25,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:25,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52e9510[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:25,814 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:25,814 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:25,815 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:25,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:25,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:25,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:25,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:25,815 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:25,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,815 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:25,816 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:25,816 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:25,816 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,816 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:25,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:25,819 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:25,819 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:25,819 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:25,820 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:25,820 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:25,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:25,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:25,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:25,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:25,823 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:25,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:25,824 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:25,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,826 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,826 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:25,826 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:25,826 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,826 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:25,826 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:25,826 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:25,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:25,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:25,826 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:25,827 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:25,827 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:25,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:25,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:25,827 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:25,827 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:25,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:25,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:25,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:25,927 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:25,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:25,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:25,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:25,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:25,931 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:25,931 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:25,931 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:25,931 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:25,931 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:25,931 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:25,932 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:25,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:25,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:25,932 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,933 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:25,933 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:25,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:25,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:25,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:25,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:25,934 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:25,934 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:25,934 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:25,935 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:25,935 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:25,935 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:25,935 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=145.36 KB heapSize=423.16 KB 2023-03-26 13:56:25,936 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:25,936 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:25,936 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:25,936 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:25,936 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:25,936 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:25,946 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326ef58c0d97cae4a8d813e44ec9c093424 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ef58c0d97cae4a8d813e44ec9c093424 2023-03-26 13:56:25,950 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1429, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/54efef2d9b794652a4d250963d717c22 2023-03-26 13:56:26,359 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1429 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/247f1fbce4a74a8983fb23dbeaa7713c 2023-03-26 13:56:26,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:26,758 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:26,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:26,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 727 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839046762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:26,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839046762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:26,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839046765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,769 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:26,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 589 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839046766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,776 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=69.33 KB at sequenceid=1429 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/7e590381548e4542ae928fcfbb9cc88d 2023-03-26 13:56:26,780 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/54efef2d9b794652a4d250963d717c22 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/54efef2d9b794652a4d250963d717c22 2023-03-26 13:56:26,783 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/54efef2d9b794652a4d250963d717c22, entries=150, sequenceid=1429, filesize=25.4 K 2023-03-26 13:56:26,784 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/247f1fbce4a74a8983fb23dbeaa7713c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/247f1fbce4a74a8983fb23dbeaa7713c 2023-03-26 13:56:26,787 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/247f1fbce4a74a8983fb23dbeaa7713c, entries=150, sequenceid=1429, filesize=11.9 K 2023-03-26 13:56:26,788 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/7e590381548e4542ae928fcfbb9cc88d as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/7e590381548e4542ae928fcfbb9cc88d 2023-03-26 13:56:26,790 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/7e590381548e4542ae928fcfbb9cc88d, entries=150, sequenceid=1429, filesize=11.9 K 2023-03-26 13:56:26,791 INFO [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~145.36 KB/148850, heapSize ~423.11 KB/433264, currentSize=40.25 KB/41220 for f904470a09c7c6a52bdab844746de048 in 856ms, sequenceid=1429, compaction requested=true 2023-03-26 13:56:26,791 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:26,791 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:26,791 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:26,791 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:26,791 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,791 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:26,791 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:26,796 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,796 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:26,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:26,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:26,796 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:26,796 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:26,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:26,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:26,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:26,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:26,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:26,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1568a396[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:26,798 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:26,798 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:26,799 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:26,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:26,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:26,799 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:26,799 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:26,799 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:26,799 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,799 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:26,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:26,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:26,800 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,800 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:26,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:26,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:26,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:26,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:26,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:26,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:26,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:26,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:26,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:26,802 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:26,802 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:26,803 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:26,803 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:26,803 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:26,804 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:26,804 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:26,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:26,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:26,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:26,804 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:26,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:26,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:26,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:26,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:26,805 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:26,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:26,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:26,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:26,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:26,806 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:26,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,809 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:26,809 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:26,809 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:26,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:26,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:26,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:26,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:26,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:26,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:26,810 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:26,810 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:26,812 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,812 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:26,812 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:26,812 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,812 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:26,812 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:26,812 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:26,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:26,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:26,868 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 13:56:26,869 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:26,869 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:26,869 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:26,869 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:26,869 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:26,869 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:26,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:26,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:26,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839046886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 595 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839046887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679838986876, max timestamp=1679838986891Num uniques 859; 2023-03-26 13:56:26,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:26,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679838986876, max timestamp=1679838986891Num uniques 859; 2023-03-26 13:56:26,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:26,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679838986876, max timestamp=1679838986891Num uniques 859; 2023-03-26 13:56:26,893 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 13:56:26,910 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:26,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:26,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:26,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:26,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:26,913 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:26,913 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:26,913 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:26,913 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:26,914 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:26,914 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:26,915 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:26,915 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:26,915 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:26,915 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:26,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:26,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:26,916 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:26,917 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:26,917 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:26,917 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:26,917 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:26,917 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:26,919 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:26,919 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:26,919 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:26,919 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:26,919 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:26,919 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:26,919 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:26,919 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:26,919 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,919 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:26,919 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:26,921 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:26,921 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:26,921 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:26,921 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:26,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:26,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:26,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67dc4d9[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:26,922 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:26,923 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:26,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:26,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839046920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,923 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:26,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:26,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:26,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:26,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:26,924 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:26,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,924 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:26,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:26,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 742 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839046922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,926 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:26,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:26,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:26,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:26,926 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:26,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:26,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:26,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:26,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:26,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:26,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:26,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:26,929 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:26,929 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:26,929 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:26,929 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:26,929 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:26,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:26,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:26,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:26,931 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:26,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:26,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:26,932 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:26,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:26,933 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:26,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:26,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:26,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:26,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:26,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:26,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:26,935 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:26,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:26,936 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:26,936 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:26,935 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:26,936 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:26,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:26,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:26,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:26,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839046994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:26,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:26,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 597 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839046994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:27,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839047025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:27,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 744 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839047025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,036 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:27,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:27,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:27,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:27,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:27,039 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:27,039 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:27,039 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:27,039 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:27,039 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,040 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,041 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,041 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:27,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:27,041 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,041 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:27,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:27,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:27,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:27,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:27,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:27,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:27,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:27,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:27,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:27,043 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:27,043 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:27,043 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:27,043 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:27,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:27,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:27,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:27,043 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:27,046 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,046 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,046 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,046 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:27,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7256a064[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:27,047 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:27,047 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,048 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,049 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:27,049 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:27,049 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:27,049 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,049 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:27,062 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,062 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:27,063 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,063 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:27,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:27,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:27,066 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:27,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:27,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:27,066 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:27,066 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:27,068 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,068 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,068 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:27,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,069 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,070 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,073 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,073 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,073 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,073 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,073 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,074 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:27,073 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:27,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:27,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,074 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:27,074 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:27,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:27,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:27,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:27,074 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:27,074 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:27,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:27,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:27,174 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:27,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:27,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:27,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:27,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:27,178 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:27,178 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:27,178 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:27,178 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:27,178 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,178 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,179 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,179 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:27,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:27,180 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,180 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:27,182 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:27,182 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:27,182 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:27,182 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:27,182 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:27,182 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:27,182 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:27,182 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:27,182 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:27,182 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:27,182 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:27,183 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:27,183 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:27,183 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,183 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:27,183 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:27,184 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,185 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,185 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:27,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:27,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a12d2b[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:27,187 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:27,187 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,188 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,188 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:27,188 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:27,188 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:27,189 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,189 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:27,194 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:27,194 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:27,194 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,194 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:27,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:27,198 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:27,198 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:27,198 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:27,198 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:27,198 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:27,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:27,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 599 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839047196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:27,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839047197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,200 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,200 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:27,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,200 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,201 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,203 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,205 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,205 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,206 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,206 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,206 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,206 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,206 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,206 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:27,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:27,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:27,206 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:27,206 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:27,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:27,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:27,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:27,208 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:27,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:27,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:27,229 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:27,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 746 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839047228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:27,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839047229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,281 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032641f8e2dc534446c9a533501d75dbe709 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032641f8e2dc534446c9a533501d75dbe709 2023-03-26 13:56:27,306 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:27,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:27,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:27,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:27,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:27,310 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:27,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:27,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:27,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:27,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,310 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,312 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,312 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:27,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:27,312 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,312 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:27,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:27,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:27,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:27,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:27,313 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:27,313 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:27,313 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:27,314 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:27,314 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:27,314 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:27,314 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:27,314 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:27,314 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:27,314 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:27,314 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:27,314 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,314 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:27,314 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:27,316 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,317 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,317 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53855ada[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:27,318 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:27,318 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,319 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,319 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:27,319 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:27,320 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:27,320 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,320 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:27,321 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,321 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:27,321 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,321 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:27,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:27,324 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:27,324 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:27,325 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:27,325 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:27,325 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:27,326 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,326 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:27,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,326 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:27,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,327 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,328 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,332 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,332 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,332 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,332 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:27,332 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:27,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:27,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:27,332 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:27,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:27,332 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:27,332 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:27,332 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:27,332 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:27,332 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,332 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:27,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:27,432 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:27,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:27,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:27,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:27,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:27,436 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:27,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:27,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:27,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:27,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,436 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,438 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:27,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:27,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,438 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:27,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:27,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:27,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:27,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:27,439 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:27,440 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:27,440 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:27,440 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:27,440 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:27,440 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:27,442 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,442 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,442 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,442 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:27,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:27,444 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:27,444 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bb6177f[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:27,445 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,445 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:27,445 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:27,445 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:27,445 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,445 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:27,447 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:27,447 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,447 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,447 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:27,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:27,451 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:27,451 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:27,451 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:27,451 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:27,451 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:27,453 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,453 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,453 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:27,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:27,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,453 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,455 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,457 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:27,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:27,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:27,457 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,458 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,458 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:27,458 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,458 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,458 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:27,458 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,458 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,458 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,458 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:27,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:27,458 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:27,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:27,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:27,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:27,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:27,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 601 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839047502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:27,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839047502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:27,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 748 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839047533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:27,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=42517] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839047533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,558 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:27,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:27,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:27,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:27,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:27,562 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:27,562 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:27,562 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:27,562 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:27,562 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,562 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,564 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,564 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:27,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:27,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:27,565 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,565 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:27,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:27,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:27,566 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:27,566 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:27,566 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:27,566 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:27,566 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:27,566 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:27,566 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:27,567 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:27,567 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:27,567 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:27,567 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:27,567 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:27,567 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,567 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:27,567 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:27,569 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,569 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,570 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,570 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:27,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ca4d5ed[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:27,571 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:27,572 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,573 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,573 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:27,573 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:27,573 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:27,573 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,573 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:27,575 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,575 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:27,575 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:27,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,575 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:27,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:27,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:27,577 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:27,577 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:27,577 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:27,577 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:27,577 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:27,582 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,582 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,582 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,583 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,584 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,589 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,589 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,589 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,589 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,589 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,589 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,589 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:27,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:27,589 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:27,589 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:27,589 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:27,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:27,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:27,589 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:27,589 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:27,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:27,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:27,686 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1440, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/53340f2d9f72474a9baca93198e810b4 2023-03-26 13:56:27,690 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:27,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:27,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:27,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:27,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:27,694 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:27,694 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:27,694 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:27,694 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:27,694 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,694 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,695 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,695 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:27,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:27,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:27,696 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,696 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:27,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:27,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:27,697 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:27,698 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:27,698 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:27,698 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:27,698 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:27,698 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:27,698 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:27,698 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:27,698 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:27,698 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:27,698 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:27,698 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,698 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:27,698 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:27,701 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,701 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,701 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,701 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:27,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:27,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76e67727[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:27,703 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:27,703 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,704 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:27,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:27,704 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:27,705 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,705 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:27,706 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:27,706 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,706 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:27,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,706 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:27,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:27,709 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:27,709 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:27,709 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:27,709 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:27,709 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:27,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:27,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,711 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:27,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:27,712 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,713 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,716 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,716 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,716 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,716 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:27,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:27,716 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:27,717 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:27,717 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:27,716 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:27,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:27,716 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,717 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:27,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:27,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:27,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:27,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:27,817 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:27,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:27,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:27,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:27,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:27,820 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:27,820 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:27,820 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:27,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:27,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,821 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,822 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:27,822 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:27,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:27,823 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,823 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:27,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:27,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:27,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:27,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:27,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:27,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:27,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:27,824 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:27,824 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:27,824 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:27,825 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:27,825 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:27,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:27,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:27,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:27,825 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:27,826 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,826 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,827 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,827 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:27,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f4ca0ab[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:27,828 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:27,828 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,829 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,830 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:27,830 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:27,830 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,830 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:27,830 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:27,832 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,832 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:27,832 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:27,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,832 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:27,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:27,835 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:27,835 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:27,835 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:27,835 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:27,835 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:27,836 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,837 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,837 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,838 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,839 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,841 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:27,841 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:27,841 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:27,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:27,841 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:27,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:27,841 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:27,842 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:27,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,842 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:27,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:27,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,842 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,942 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:27,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:27,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:27,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:27,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:27,946 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:27,946 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:27,946 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:27,946 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:27,946 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,946 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,947 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:27,947 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:27,948 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,948 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:27,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:27,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:27,949 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:27,949 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:27,949 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:27,949 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:27,949 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,950 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:27,950 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:27,952 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,952 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,952 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,952 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:27,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:27,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44a53d2[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:27,954 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:27,954 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,955 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,955 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:27,955 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:27,955 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:27,955 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,955 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:27,956 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,956 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:27,957 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:27,957 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:27,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:27,959 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:27,959 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:27,959 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:27,959 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:27,959 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:27,961 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:27,961 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,961 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:27,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:27,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:27,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:27,961 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:27,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,963 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:27,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:27,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,966 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,966 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,966 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:27,966 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:27,966 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:27,966 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:27,966 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:27,966 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,966 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:27,966 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:27,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:27,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:27,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:27,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:27,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:27,967 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:27,967 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:27,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:27,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:28,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:28,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46358 deadline: 1679839048005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:28,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 603 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46380 deadline: 1679839048006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:28,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 750 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46364 deadline: 1679839048037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 13:56:28,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=42517] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46350 deadline: 1679839048038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,067 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:28,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:28,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 13:56:28,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 13:56:28,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 13:56:28,070 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 13:56:28,070 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:28,070 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 13:56:28,070 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 13:56:28,070 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:28,070 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:28,072 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:28,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:28,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,072 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 13:56:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:28,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 13:56:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:28,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 13:56:28,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 13:56:28,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 13:56:28,073 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 13:56:28,073 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 13:56:28,073 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 13:56:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 13:56:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 13:56:28,074 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:28,074 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. started... 2023-03-26 13:56:28,074 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. as already flushing 2023-03-26 13:56:28,074 DEBUG [rs(jenkins-hbase4.apache.org,42517,1679838814721)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 13:56:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 13:56:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 13:56:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 13:56:28,075 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:28,075 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:28,076 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:28,076 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 13:56:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:28,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,42517,1679838814721' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 13:56:28,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e9bf50b[Count = 0] remaining members to acquire global barrier 2023-03-26 13:56:28,078 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 13:56:28,078 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:28,079 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:28,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:28,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 13:56:28,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 13:56:28,079 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,42517,1679838814721' in zk 2023-03-26 13:56:28,079 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,079 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 13:56:28,081 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 13:56:28,081 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:28,081 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:28,081 DEBUG [member: 'jenkins-hbase4.apache.org,42517,1679838814721' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 13:56:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,42517,1679838814721': 2023-03-26 13:56:28,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,42517,1679838814721' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 13:56:28,083 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 13:56:28,083 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 13:56:28,083 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 13:56:28,083 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 13:56:28,083 INFO [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 13:56:28,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:28,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:28,085 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:28,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 13:56:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 13:56:28,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 13:56:28,086 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 13:56:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:28,087 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 13:56:28,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 13:56:28,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:28,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 13:56:28,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:28,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:28,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 13:56:28,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 13:56:28,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 13:56:28,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 13:56:28,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 13:56:28,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 13:56:28,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 13:56:28,089 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 13:56:28,089 DEBUG [(jenkins-hbase4.apache.org,45485,1679838813504)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 13:56:28,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 13:56:28,090 DEBUG [Thread-4361] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 13:56:28,090 DEBUG [Thread-4361] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 13:56:28,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 13:56:28,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 13:56:28,097 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1440 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/c445e5a8e7384f96abd93a5022e51670 2023-03-26 13:56:28,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1440 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/5bddb7c33e20468485b757f7269fdfd4 2023-03-26 13:56:28,107 DEBUG [Thread-4368] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77f9399b to 127.0.0.1:54332 2023-03-26 13:56:28,107 DEBUG [Thread-4368] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:28,107 DEBUG [Thread-4370] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x430757a6 to 127.0.0.1:54332 2023-03-26 13:56:28,108 DEBUG [Thread-4370] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:28,109 DEBUG [Thread-4362] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x156e4ba6 to 127.0.0.1:54332 2023-03-26 13:56:28,109 DEBUG [Thread-4362] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:28,110 DEBUG [Thread-4364] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x785e366b to 127.0.0.1:54332 2023-03-26 13:56:28,110 DEBUG [Thread-4364] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:28,110 DEBUG [Thread-4366] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5fdfef3b to 127.0.0.1:54332 2023-03-26 13:56:28,110 DEBUG [Thread-4366] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:28,114 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/53340f2d9f72474a9baca93198e810b4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/53340f2d9f72474a9baca93198e810b4 2023-03-26 13:56:28,117 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/53340f2d9f72474a9baca93198e810b4, entries=200, sequenceid=1440, filesize=32.3 K 2023-03-26 13:56:28,117 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/c445e5a8e7384f96abd93a5022e51670 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/c445e5a8e7384f96abd93a5022e51670 2023-03-26 13:56:28,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/c445e5a8e7384f96abd93a5022e51670, entries=150, sequenceid=1440, filesize=11.9 K 2023-03-26 13:56:28,121 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/5bddb7c33e20468485b757f7269fdfd4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5bddb7c33e20468485b757f7269fdfd4 2023-03-26 13:56:28,123 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5bddb7c33e20468485b757f7269fdfd4, entries=150, sequenceid=1440, filesize=11.9 K 2023-03-26 13:56:28,124 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for f904470a09c7c6a52bdab844746de048 in 1256ms, sequenceid=1440, compaction requested=true 2023-03-26 13:56:28,124 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:28,124 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:28,124 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:28,124 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 13:56:28,124 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 13:56:28,124 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:28,125 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105402 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 13:56:28,125 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/A is initiating minor compaction (all files) 2023-03-26 13:56:28,125 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/A in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:28,125 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/fcb88183ed6a41a3aeee9e580a06f3fb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/872870fdf8c6441b8e1158b77c9e157c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/54efef2d9b794652a4d250963d717c22, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/53340f2d9f72474a9baca93198e810b4] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=102.9 K 2023-03-26 13:56:28,125 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59678 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 13:56:28,125 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/B is initiating minor compaction (all files) 2023-03-26 13:56:28,125 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/B in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:28,125 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/e34190cebba8417196ca6c31bd6396ef, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/113887428aca49c6b9a69b8cf85b2dc9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/38cdcf56c8d04f57919aa55acf7777a4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/247f1fbce4a74a8983fb23dbeaa7713c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/c445e5a8e7384f96abd93a5022e51670] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=58.3 K 2023-03-26 13:56:28,125 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting fcb88183ed6a41a3aeee9e580a06f3fb, keycount=150, bloomtype=ROW, size=26.6 K, encoding=NONE, compression=NONE, seqNum=1383, earliestPutTs=1679838983982 2023-03-26 13:56:28,126 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting e34190cebba8417196ca6c31bd6396ef, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=1344, earliestPutTs=1679838983931 2023-03-26 13:56:28,126 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 872870fdf8c6441b8e1158b77c9e157c, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=1393, earliestPutTs=1679838984003 2023-03-26 13:56:28,126 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 113887428aca49c6b9a69b8cf85b2dc9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1383, earliestPutTs=1679838983982 2023-03-26 13:56:28,126 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 54efef2d9b794652a4d250963d717c22, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1429, earliestPutTs=1679838984623 2023-03-26 13:56:28,126 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 38cdcf56c8d04f57919aa55acf7777a4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1393, earliestPutTs=1679838984003 2023-03-26 13:56:28,126 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] compactions.Compactor(207): Compacting 53340f2d9f72474a9baca93198e810b4, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1440, earliestPutTs=1679838986764 2023-03-26 13:56:28,126 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 247f1fbce4a74a8983fb23dbeaa7713c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1429, earliestPutTs=1679838984623 2023-03-26 13:56:28,127 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting c445e5a8e7384f96abd93a5022e51670, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1440, earliestPutTs=1679838986764 2023-03-26 13:56:28,133 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#B#compaction#703 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:28,134 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#A#compaction#702 average throughput is 9.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:28,190 DEBUG [Thread-4361] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 13:56:28,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 13:56:28,541 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/00534be0a532448b9aaa9a8e27f6eb54 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/00534be0a532448b9aaa9a8e27f6eb54 2023-03-26 13:56:28,544 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f904470a09c7c6a52bdab844746de048/B of f904470a09c7c6a52bdab844746de048 into 00534be0a532448b9aaa9a8e27f6eb54(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:28,544 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:28,544 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/B, priority=11, startTime=1679838988124; duration=0sec 2023-03-26 13:56:28,544 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 13:56:28,545 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 13:56:28,545 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59678 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 13:56:28,546 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1912): f904470a09c7c6a52bdab844746de048/C is initiating minor compaction (all files) 2023-03-26 13:56:28,546 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f904470a09c7c6a52bdab844746de048/C in TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:28,546 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/4377046344354ee493e82e1d98f7d644, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/4d303961e5eb43afa951f75f2f00d00a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/37f40c8fa6734189818d5a94ec97ae01, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/7e590381548e4542ae928fcfbb9cc88d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5bddb7c33e20468485b757f7269fdfd4] into tmpdir=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp, totalSize=58.3 K 2023-03-26 13:56:28,546 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 4377046344354ee493e82e1d98f7d644, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=1344, earliestPutTs=1679838983931 2023-03-26 13:56:28,546 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 4d303961e5eb43afa951f75f2f00d00a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1383, earliestPutTs=1679838983982 2023-03-26 13:56:28,547 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 37f40c8fa6734189818d5a94ec97ae01, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=1393, earliestPutTs=1679838984003 2023-03-26 13:56:28,547 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 7e590381548e4542ae928fcfbb9cc88d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1429, earliestPutTs=1679838984623 2023-03-26 13:56:28,547 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] compactions.Compactor(207): Compacting 5bddb7c33e20468485b757f7269fdfd4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1440, earliestPutTs=1679838986764 2023-03-26 13:56:28,554 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] throttle.PressureAwareThroughputController(145): f904470a09c7c6a52bdab844746de048#C#compaction#704 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 13:56:28,560 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/d3e3ec2eef7d4bbea1e4847faf8421ba as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/d3e3ec2eef7d4bbea1e4847faf8421ba 2023-03-26 13:56:28,563 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in f904470a09c7c6a52bdab844746de048/C of f904470a09c7c6a52bdab844746de048 into d3e3ec2eef7d4bbea1e4847faf8421ba(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 13:56:28,563 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:28,563 INFO [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/C, priority=11, startTime=1679838988124; duration=0sec 2023-03-26 13:56:28,563 DEBUG [RS:0;jenkins-hbase4:42517-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:29,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=42517] regionserver.HRegion(9158): Flush requested on f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:29,012 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-26 13:56:29,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:29,012 DEBUG [Thread-4355] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x526f2221 to 127.0.0.1:54332 2023-03-26 13:56:29,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:29,012 DEBUG [Thread-4355] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:29,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:29,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:29,012 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:29,012 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 13:56:29,020 DEBUG [Thread-4353] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x19c1e3ee to 127.0.0.1:54332 2023-03-26 13:56:29,020 DEBUG [Thread-4353] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:29,046 DEBUG [Thread-4357] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x17cf2e3f to 127.0.0.1:54332 2023-03-26 13:56:29,046 DEBUG [Thread-4357] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:29,049 DEBUG [Thread-4351] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a320817 to 127.0.0.1:54332 2023-03-26 13:56:29,049 DEBUG [Thread-4351] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:29,348 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/6a5929d7379842c4a3642d75a1e1835c as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/6a5929d7379842c4a3642d75a1e1835c 2023-03-26 13:56:29,351 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f904470a09c7c6a52bdab844746de048/A of f904470a09c7c6a52bdab844746de048 into 6a5929d7379842c4a3642d75a1e1835c(size=26.8 K), total size for store is 26.8 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-26 13:56:29,351 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:29,351 INFO [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048., storeName=f904470a09c7c6a52bdab844746de048/A, priority=12, startTime=1679838988124; duration=1sec 2023-03-26 13:56:29,352 DEBUG [RS:0;jenkins-hbase4:42517-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 13:56:29,422 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326f1e7838f89974be88251a91f6660c9fc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f1e7838f89974be88251a91f6660c9fc 2023-03-26 13:56:29,826 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1478, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/9a587fa13aaa4318a4f5f5537706c58e 2023-03-26 13:56:30,234 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1478 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/5c86b0542d1a460b87f38e08eafecee3 2023-03-26 13:56:30,642 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=1478 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/17ef05bbf1294ba680c25d412fa594ff 2023-03-26 13:56:30,646 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/9a587fa13aaa4318a4f5f5537706c58e as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/9a587fa13aaa4318a4f5f5537706c58e 2023-03-26 13:56:30,648 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/9a587fa13aaa4318a4f5f5537706c58e, entries=150, sequenceid=1478, filesize=25.4 K 2023-03-26 13:56:30,649 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/5c86b0542d1a460b87f38e08eafecee3 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/5c86b0542d1a460b87f38e08eafecee3 2023-03-26 13:56:30,651 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/5c86b0542d1a460b87f38e08eafecee3, entries=150, sequenceid=1478, filesize=11.9 K 2023-03-26 13:56:30,652 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/17ef05bbf1294ba680c25d412fa594ff as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/17ef05bbf1294ba680c25d412fa594ff 2023-03-26 13:56:30,655 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/17ef05bbf1294ba680c25d412fa594ff, entries=150, sequenceid=1478, filesize=11.9 K 2023-03-26 13:56:30,655 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=20.13 KB/20610 for f904470a09c7c6a52bdab844746de048 in 1643ms, sequenceid=1478, compaction requested=false 2023-03-26 13:56:30,655 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:31,744 DEBUG [Thread-4359] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3420e54c to 127.0.0.1:54332 2023-03-26 13:56:31,744 DEBUG [Thread-4359] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:31,744 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-26 13:56:31,745 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 617 2023-03-26 13:56:31,745 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 460 2023-03-26 13:56:31,745 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 127 2023-03-26 13:56:31,745 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 111 2023-03-26 13:56:31,745 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(394): wrote 38 2023-03-26 13:56:31,745 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-26 13:56:31,745 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(398): read 4335 2023-03-26 13:56:31,745 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(398): read 4331 2023-03-26 13:56:31,745 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(398): read 4340 2023-03-26 13:56:31,745 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(398): read 4326 2023-03-26 13:56:31,745 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(398): read 4327 2023-03-26 13:56:31,745 INFO [Listener at localhost/41531] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-26 13:56:31,745 INFO [Listener at localhost/41531] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-26 13:56:31,745 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x55d0f313 to 127.0.0.1:54332 2023-03-26 13:56:31,745 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:31,746 INFO [Listener at localhost/41531] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-26 13:56:31,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-26 13:56:31,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-26 13:56:31,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-26 13:56:31,749 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838991749"}]},"ts":"1679838991749"} 2023-03-26 13:56:31,750 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-26 13:56:31,751 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-26 13:56:31,752 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f904470a09c7c6a52bdab844746de048, UNASSIGN}] 2023-03-26 13:56:31,753 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f904470a09c7c6a52bdab844746de048, UNASSIGN 2023-03-26 13:56:31,753 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=f904470a09c7c6a52bdab844746de048, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:31,754 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838991753"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679838991753"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679838991753"}]},"ts":"1679838991753"} 2023-03-26 13:56:31,754 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721}] 2023-03-26 13:56:31,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-26 13:56:31,906 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:31,906 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing f904470a09c7c6a52bdab844746de048, disabling compactions & flushes 2023-03-26 13:56:31,906 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:31,906 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:31,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. after waiting 0 ms 2023-03-26 13:56:31,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:31,907 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing f904470a09c7c6a52bdab844746de048 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-26 13:56:31,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=A 2023-03-26 13:56:31,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:31,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=B 2023-03-26 13:56:31,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:31,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f904470a09c7c6a52bdab844746de048, store=C 2023-03-26 13:56:31,907 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 13:56:31,916 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326ba3b171520064a0eb74175b394dfb9ae to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ba3b171520064a0eb74175b394dfb9ae 2023-03-26 13:56:32,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-26 13:56:32,320 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1486, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/780a30fea3a7459b9a6828bd795c6100 2023-03-26 13:56:32,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-26 13:56:32,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1486 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/7207c61e929e47028d4aab61359ff494 2023-03-26 13:56:32,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-26 13:56:33,136 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1486 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/697f292327b8472faba000f047227e7b 2023-03-26 13:56:33,139 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/A/780a30fea3a7459b9a6828bd795c6100 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/780a30fea3a7459b9a6828bd795c6100 2023-03-26 13:56:33,141 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/780a30fea3a7459b9a6828bd795c6100, entries=150, sequenceid=1486, filesize=25.4 K 2023-03-26 13:56:33,142 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/B/7207c61e929e47028d4aab61359ff494 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/7207c61e929e47028d4aab61359ff494 2023-03-26 13:56:33,145 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/7207c61e929e47028d4aab61359ff494, entries=150, sequenceid=1486, filesize=11.9 K 2023-03-26 13:56:33,145 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/.tmp/C/697f292327b8472faba000f047227e7b as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/697f292327b8472faba000f047227e7b 2023-03-26 13:56:33,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/697f292327b8472faba000f047227e7b, entries=150, sequenceid=1486, filesize=11.9 K 2023-03-26 13:56:33,148 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for f904470a09c7c6a52bdab844746de048 in 1241ms, sequenceid=1486, compaction requested=true 2023-03-26 13:56:33,172 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/90c2a16f4af84d9db1e0bb2bb45a144c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/8ac1b81b14e04108a8537bbc0811eb22, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/0ad62a672be04218ba8805695786ad7a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/0993abfab80b410cbed406edc1a56926, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/1e424e85b5a44957adf6c8327ec5c781, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/6d96e855229d4d69b99655ab956e85fd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/a4d8c6e1c4174296bc54e44b9d0ace30, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/6aee5876e3a246ed9628a2d3e8b747fd, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/42fc1c46c1814651bc29473e0a7273db, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/439eafdc7d6a4360bb0a1a371681af6a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/e2c67069d15e4d4ebfff483b04823381, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/68eb1c0dade24dbb9eeed2f9de630e15, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/faa103eddba3441e8331cfd71d1851d7, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/e1aea2a8e93c46c8a2e7394f8e52de01, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/c637826cdead4dbc8f97a19019d69b20, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/0918c56d03b54de7a4c4e966cba08d4f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/29e006ab97b543f6b16e209667d7598e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/82bf935d2db94de7a07100b60b29137d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/12b61e62b99b469fae365cca2fe4bea2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/c719f4fcd9fd456587bc48b7c6cef579, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/4590cff7c622414b8752dfb73ba71a58, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/8daf3bd1b5454ec3a28da6bbaf842336, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/478de32ff85f48b49eeb1b1335a1c783, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/54da4f30389e44f8a068b8b1d23a8aed, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/04a2b7185a5c4727a42c62e84e1916ad, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/202cbfcaebca444cb921ad780065d964, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/efdf28276a904972a3e407b169686bc0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/fff8943ea46a4b54bdea14bf68a396f2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/51218eba893043d7b7b8a96a024f48f9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/79107eb01d3f4804b0a08b1ab8b636f4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/a5d64006bdbf422b818b10445df8cf04, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/cd5a33cda9d44e8495937913b288cc22, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/71618c6d257f41519a6cef2e38f719b1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/add1320c10bb4e7ea8aa100798464170, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/9ddec078aa52461880fed76f6d87e4d6, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/917d4ebc10d54df8b9d6c409657380f1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/fcb88183ed6a41a3aeee9e580a06f3fb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/872870fdf8c6441b8e1158b77c9e157c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/54efef2d9b794652a4d250963d717c22, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/53340f2d9f72474a9baca93198e810b4] to archive 2023-03-26 13:56:33,173 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 13:56:33,174 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/90c2a16f4af84d9db1e0bb2bb45a144c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/90c2a16f4af84d9db1e0bb2bb45a144c 2023-03-26 13:56:33,175 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/8ac1b81b14e04108a8537bbc0811eb22 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/8ac1b81b14e04108a8537bbc0811eb22 2023-03-26 13:56:33,176 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/0ad62a672be04218ba8805695786ad7a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/0ad62a672be04218ba8805695786ad7a 2023-03-26 13:56:33,177 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/0993abfab80b410cbed406edc1a56926 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/0993abfab80b410cbed406edc1a56926 2023-03-26 13:56:33,178 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/1e424e85b5a44957adf6c8327ec5c781 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/1e424e85b5a44957adf6c8327ec5c781 2023-03-26 13:56:33,178 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/6d96e855229d4d69b99655ab956e85fd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/6d96e855229d4d69b99655ab956e85fd 2023-03-26 13:56:33,179 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/a4d8c6e1c4174296bc54e44b9d0ace30 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/a4d8c6e1c4174296bc54e44b9d0ace30 2023-03-26 13:56:33,180 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/6aee5876e3a246ed9628a2d3e8b747fd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/6aee5876e3a246ed9628a2d3e8b747fd 2023-03-26 13:56:33,181 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/42fc1c46c1814651bc29473e0a7273db to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/42fc1c46c1814651bc29473e0a7273db 2023-03-26 13:56:33,182 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/439eafdc7d6a4360bb0a1a371681af6a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/439eafdc7d6a4360bb0a1a371681af6a 2023-03-26 13:56:33,183 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/e2c67069d15e4d4ebfff483b04823381 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/e2c67069d15e4d4ebfff483b04823381 2023-03-26 13:56:33,184 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/68eb1c0dade24dbb9eeed2f9de630e15 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/68eb1c0dade24dbb9eeed2f9de630e15 2023-03-26 13:56:33,185 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/faa103eddba3441e8331cfd71d1851d7 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/faa103eddba3441e8331cfd71d1851d7 2023-03-26 13:56:33,186 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/e1aea2a8e93c46c8a2e7394f8e52de01 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/e1aea2a8e93c46c8a2e7394f8e52de01 2023-03-26 13:56:33,186 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/c637826cdead4dbc8f97a19019d69b20 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/c637826cdead4dbc8f97a19019d69b20 2023-03-26 13:56:33,187 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/0918c56d03b54de7a4c4e966cba08d4f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/0918c56d03b54de7a4c4e966cba08d4f 2023-03-26 13:56:33,188 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/29e006ab97b543f6b16e209667d7598e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/29e006ab97b543f6b16e209667d7598e 2023-03-26 13:56:33,188 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/82bf935d2db94de7a07100b60b29137d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/82bf935d2db94de7a07100b60b29137d 2023-03-26 13:56:33,189 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/12b61e62b99b469fae365cca2fe4bea2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/12b61e62b99b469fae365cca2fe4bea2 2023-03-26 13:56:33,190 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/c719f4fcd9fd456587bc48b7c6cef579 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/c719f4fcd9fd456587bc48b7c6cef579 2023-03-26 13:56:33,191 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/4590cff7c622414b8752dfb73ba71a58 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/4590cff7c622414b8752dfb73ba71a58 2023-03-26 13:56:33,192 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/8daf3bd1b5454ec3a28da6bbaf842336 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/8daf3bd1b5454ec3a28da6bbaf842336 2023-03-26 13:56:33,193 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/478de32ff85f48b49eeb1b1335a1c783 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/478de32ff85f48b49eeb1b1335a1c783 2023-03-26 13:56:33,194 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/54da4f30389e44f8a068b8b1d23a8aed to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/54da4f30389e44f8a068b8b1d23a8aed 2023-03-26 13:56:33,194 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/04a2b7185a5c4727a42c62e84e1916ad to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/04a2b7185a5c4727a42c62e84e1916ad 2023-03-26 13:56:33,195 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/202cbfcaebca444cb921ad780065d964 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/202cbfcaebca444cb921ad780065d964 2023-03-26 13:56:33,196 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/efdf28276a904972a3e407b169686bc0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/efdf28276a904972a3e407b169686bc0 2023-03-26 13:56:33,197 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/fff8943ea46a4b54bdea14bf68a396f2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/fff8943ea46a4b54bdea14bf68a396f2 2023-03-26 13:56:33,198 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/51218eba893043d7b7b8a96a024f48f9 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/51218eba893043d7b7b8a96a024f48f9 2023-03-26 13:56:33,199 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/79107eb01d3f4804b0a08b1ab8b636f4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/79107eb01d3f4804b0a08b1ab8b636f4 2023-03-26 13:56:33,199 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/a5d64006bdbf422b818b10445df8cf04 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/a5d64006bdbf422b818b10445df8cf04 2023-03-26 13:56:33,200 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/cd5a33cda9d44e8495937913b288cc22 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/cd5a33cda9d44e8495937913b288cc22 2023-03-26 13:56:33,201 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/71618c6d257f41519a6cef2e38f719b1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/71618c6d257f41519a6cef2e38f719b1 2023-03-26 13:56:33,202 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/add1320c10bb4e7ea8aa100798464170 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/add1320c10bb4e7ea8aa100798464170 2023-03-26 13:56:33,203 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/9ddec078aa52461880fed76f6d87e4d6 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/9ddec078aa52461880fed76f6d87e4d6 2023-03-26 13:56:33,204 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/917d4ebc10d54df8b9d6c409657380f1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/917d4ebc10d54df8b9d6c409657380f1 2023-03-26 13:56:33,204 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/fcb88183ed6a41a3aeee9e580a06f3fb to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/fcb88183ed6a41a3aeee9e580a06f3fb 2023-03-26 13:56:33,205 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/872870fdf8c6441b8e1158b77c9e157c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/872870fdf8c6441b8e1158b77c9e157c 2023-03-26 13:56:33,207 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/54efef2d9b794652a4d250963d717c22 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/54efef2d9b794652a4d250963d717c22 2023-03-26 13:56:33,208 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/53340f2d9f72474a9baca93198e810b4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/53340f2d9f72474a9baca93198e810b4 2023-03-26 13:56:33,211 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1e7265ad98b34235a81b53dce03ed87d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1e2b333f69e64de48dd0c3180bdcc576, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/433907e55f86483582fa4b23cfc249ee, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/4af011ec655d44aa9360182413b3850e, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/f128f6749e954bbab4f6564e311d4aad, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/fbbd03c871144de4895e20978ebbf576, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/53eb9150507c4541953a4cfd1043eda0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/84a002c15dba4981ac1d65b94554a8bb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/45c74fbd7d9540cfa34b36dc866f99b1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/d8eb7e553aef4cc89784022a8e1c3e3c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/91ad5684e0bb4e188f1e34b1b92d3a7c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/54f4bfff636845a19f1efc1dfbff70b8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/661c919c39bd4b048be6bd60de91b8dc, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/b75d47b92f6344dea23ffbdf9a4516b5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/af5b1d5e585841a2a9058b55b843bb30, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1636dcf0a7214d689c1e06298a48a9cb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/bac998b9ce2f45a3bd58aa843d1b43be, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/201036b34a0a4d37b385713c010b6416, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/af9b17e4c2d944ea8fdaa36e14644c5f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/b6410a487db54cbdaeefedc05669dde2, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/539fea6ceb3446bcb01cdada2536760f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/3731a9ed513d43f3958984221b023a9a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/301d14ff980045e78380989fb48c898b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/808770745f8b4d08a9afd1ed9d11e1e0, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1d4e38503fc74fdf89b8ff9a30f61d93, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/074027c947aa4867b7a6cb93103612ef, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/45ef37e13b37403593fe0e9018cef616, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/f9b68eaa74434341a115170365307d8f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/9b627a0acd9b485295d73d8d22ef5654, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/cbfd0f4c689e4f2889fcb43e883691bb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/32d52bb015804e28b0de93c0333942c4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/99d2ff895b1d4999a97c223f0e1c37b3, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/dfaa425c198e4de5a60fe0cf1afeb38f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/3901a9ce446b452298007bb76396c3f8, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/e34190cebba8417196ca6c31bd6396ef, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/ae3f03375c8b42028cb9fc3d25caaccf, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/113887428aca49c6b9a69b8cf85b2dc9, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/38cdcf56c8d04f57919aa55acf7777a4, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/247f1fbce4a74a8983fb23dbeaa7713c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/c445e5a8e7384f96abd93a5022e51670] to archive 2023-03-26 13:56:33,211 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 13:56:33,212 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1e7265ad98b34235a81b53dce03ed87d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1e7265ad98b34235a81b53dce03ed87d 2023-03-26 13:56:33,213 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1e2b333f69e64de48dd0c3180bdcc576 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1e2b333f69e64de48dd0c3180bdcc576 2023-03-26 13:56:33,214 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/433907e55f86483582fa4b23cfc249ee to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/433907e55f86483582fa4b23cfc249ee 2023-03-26 13:56:33,215 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/4af011ec655d44aa9360182413b3850e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/4af011ec655d44aa9360182413b3850e 2023-03-26 13:56:33,215 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/f128f6749e954bbab4f6564e311d4aad to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/f128f6749e954bbab4f6564e311d4aad 2023-03-26 13:56:33,216 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/fbbd03c871144de4895e20978ebbf576 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/fbbd03c871144de4895e20978ebbf576 2023-03-26 13:56:33,217 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/53eb9150507c4541953a4cfd1043eda0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/53eb9150507c4541953a4cfd1043eda0 2023-03-26 13:56:33,217 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/84a002c15dba4981ac1d65b94554a8bb to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/84a002c15dba4981ac1d65b94554a8bb 2023-03-26 13:56:33,218 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/45c74fbd7d9540cfa34b36dc866f99b1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/45c74fbd7d9540cfa34b36dc866f99b1 2023-03-26 13:56:33,219 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/d8eb7e553aef4cc89784022a8e1c3e3c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/d8eb7e553aef4cc89784022a8e1c3e3c 2023-03-26 13:56:33,220 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/91ad5684e0bb4e188f1e34b1b92d3a7c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/91ad5684e0bb4e188f1e34b1b92d3a7c 2023-03-26 13:56:33,220 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/54f4bfff636845a19f1efc1dfbff70b8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/54f4bfff636845a19f1efc1dfbff70b8 2023-03-26 13:56:33,221 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/661c919c39bd4b048be6bd60de91b8dc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/661c919c39bd4b048be6bd60de91b8dc 2023-03-26 13:56:33,222 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/b75d47b92f6344dea23ffbdf9a4516b5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/b75d47b92f6344dea23ffbdf9a4516b5 2023-03-26 13:56:33,222 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/af5b1d5e585841a2a9058b55b843bb30 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/af5b1d5e585841a2a9058b55b843bb30 2023-03-26 13:56:33,223 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1636dcf0a7214d689c1e06298a48a9cb to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1636dcf0a7214d689c1e06298a48a9cb 2023-03-26 13:56:33,224 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/bac998b9ce2f45a3bd58aa843d1b43be to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/bac998b9ce2f45a3bd58aa843d1b43be 2023-03-26 13:56:33,225 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/201036b34a0a4d37b385713c010b6416 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/201036b34a0a4d37b385713c010b6416 2023-03-26 13:56:33,225 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/af9b17e4c2d944ea8fdaa36e14644c5f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/af9b17e4c2d944ea8fdaa36e14644c5f 2023-03-26 13:56:33,226 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/b6410a487db54cbdaeefedc05669dde2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/b6410a487db54cbdaeefedc05669dde2 2023-03-26 13:56:33,227 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/539fea6ceb3446bcb01cdada2536760f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/539fea6ceb3446bcb01cdada2536760f 2023-03-26 13:56:33,227 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/3731a9ed513d43f3958984221b023a9a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/3731a9ed513d43f3958984221b023a9a 2023-03-26 13:56:33,228 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/301d14ff980045e78380989fb48c898b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/301d14ff980045e78380989fb48c898b 2023-03-26 13:56:33,229 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/808770745f8b4d08a9afd1ed9d11e1e0 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/808770745f8b4d08a9afd1ed9d11e1e0 2023-03-26 13:56:33,230 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1d4e38503fc74fdf89b8ff9a30f61d93 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/1d4e38503fc74fdf89b8ff9a30f61d93 2023-03-26 13:56:33,230 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/074027c947aa4867b7a6cb93103612ef to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/074027c947aa4867b7a6cb93103612ef 2023-03-26 13:56:33,231 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/45ef37e13b37403593fe0e9018cef616 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/45ef37e13b37403593fe0e9018cef616 2023-03-26 13:56:33,232 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/f9b68eaa74434341a115170365307d8f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/f9b68eaa74434341a115170365307d8f 2023-03-26 13:56:33,233 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/9b627a0acd9b485295d73d8d22ef5654 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/9b627a0acd9b485295d73d8d22ef5654 2023-03-26 13:56:33,233 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/cbfd0f4c689e4f2889fcb43e883691bb to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/cbfd0f4c689e4f2889fcb43e883691bb 2023-03-26 13:56:33,234 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/32d52bb015804e28b0de93c0333942c4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/32d52bb015804e28b0de93c0333942c4 2023-03-26 13:56:33,235 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/99d2ff895b1d4999a97c223f0e1c37b3 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/99d2ff895b1d4999a97c223f0e1c37b3 2023-03-26 13:56:33,236 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/dfaa425c198e4de5a60fe0cf1afeb38f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/dfaa425c198e4de5a60fe0cf1afeb38f 2023-03-26 13:56:33,237 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/3901a9ce446b452298007bb76396c3f8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/3901a9ce446b452298007bb76396c3f8 2023-03-26 13:56:33,237 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/e34190cebba8417196ca6c31bd6396ef to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/e34190cebba8417196ca6c31bd6396ef 2023-03-26 13:56:33,238 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/ae3f03375c8b42028cb9fc3d25caaccf to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/ae3f03375c8b42028cb9fc3d25caaccf 2023-03-26 13:56:33,239 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/113887428aca49c6b9a69b8cf85b2dc9 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/113887428aca49c6b9a69b8cf85b2dc9 2023-03-26 13:56:33,240 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/38cdcf56c8d04f57919aa55acf7777a4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/38cdcf56c8d04f57919aa55acf7777a4 2023-03-26 13:56:33,240 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/247f1fbce4a74a8983fb23dbeaa7713c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/247f1fbce4a74a8983fb23dbeaa7713c 2023-03-26 13:56:33,241 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/c445e5a8e7384f96abd93a5022e51670 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/c445e5a8e7384f96abd93a5022e51670 2023-03-26 13:56:33,244 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/1d82424a4ddf466787044d6ba1570b6a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/02cddce57c2b4004bda9351afc72f0da, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/b20f49074f5049cfa424062eddd93c35, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5062db6f30414173bd1d3822a1c577fe, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/d222394ab046470f93ab07340fa2a8f7, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/c06cf98dd9754d5db6b5a999c88b335b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5cd34c17d1124e03892cf184b7ecfb5b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/df30b84ccabf43c0bc147c20ecd25794, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/66e4352320ce4b68b8383a950f941729, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/6e2f8f86530b44b292e5a106604e4c88, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/504a9b4a505641ad933963bae766d958, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/7e16d5539a624e918b35c44409d7112a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/592b3b19d7eb4e1090097d80a0c03c77, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/13537a9fdd044af0b15ceaab62013c4f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/b987176d581d4be092f8958fee226836, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/ce45cc607feb447580e0f9415641f53f, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/684a473d246b4fbb8b3f9d0996d32e43, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/68a0cefd47c949cd992ef3611c2cbf90, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/795b3597ae0e4b0ba5e6285e1cebb90b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/15cc4feea2304544953a476e27e1f90a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/f5c9a26cddd24fbf9950540e374697ed, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/aaef2502f97444d7bd46de64dca2be80, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/1308f8b1e31a473abdeb636f10e5c77c, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/c8e812fc39c54d83816d095dd99c86ab, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/a7fb8a70234a4cd08f7c84179546b1ea, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/246e7b75270c45878fda2544e269e9c5, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5cc807e528bf47559b3da507a9a8b5b1, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/fdba608112264d99b20a35c81e4e801b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/315cb44a48a247b8a05e0241650c373b, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/49a708e6928d43bf8a2f1a8f58abee24, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/35c7910ffb694d0c954c8716a7da2912, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/e2476138db9b4111916aea48abf77b23, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/04ce652266cc44248c9f965814de4abf, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/96cac0dbb3e3407fae9395dd56f62ceb, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/4377046344354ee493e82e1d98f7d644, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/d9071412486c408eb5fc777fadfa8d11, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/4d303961e5eb43afa951f75f2f00d00a, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/37f40c8fa6734189818d5a94ec97ae01, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/7e590381548e4542ae928fcfbb9cc88d, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5bddb7c33e20468485b757f7269fdfd4] to archive 2023-03-26 13:56:33,245 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 13:56:33,246 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/1d82424a4ddf466787044d6ba1570b6a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/1d82424a4ddf466787044d6ba1570b6a 2023-03-26 13:56:33,247 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/02cddce57c2b4004bda9351afc72f0da to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/02cddce57c2b4004bda9351afc72f0da 2023-03-26 13:56:33,247 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/b20f49074f5049cfa424062eddd93c35 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/b20f49074f5049cfa424062eddd93c35 2023-03-26 13:56:33,248 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5062db6f30414173bd1d3822a1c577fe to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5062db6f30414173bd1d3822a1c577fe 2023-03-26 13:56:33,249 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/d222394ab046470f93ab07340fa2a8f7 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/d222394ab046470f93ab07340fa2a8f7 2023-03-26 13:56:33,250 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/c06cf98dd9754d5db6b5a999c88b335b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/c06cf98dd9754d5db6b5a999c88b335b 2023-03-26 13:56:33,250 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5cd34c17d1124e03892cf184b7ecfb5b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5cd34c17d1124e03892cf184b7ecfb5b 2023-03-26 13:56:33,251 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/df30b84ccabf43c0bc147c20ecd25794 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/df30b84ccabf43c0bc147c20ecd25794 2023-03-26 13:56:33,252 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/66e4352320ce4b68b8383a950f941729 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/66e4352320ce4b68b8383a950f941729 2023-03-26 13:56:33,253 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/6e2f8f86530b44b292e5a106604e4c88 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/6e2f8f86530b44b292e5a106604e4c88 2023-03-26 13:56:33,253 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/504a9b4a505641ad933963bae766d958 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/504a9b4a505641ad933963bae766d958 2023-03-26 13:56:33,254 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/7e16d5539a624e918b35c44409d7112a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/7e16d5539a624e918b35c44409d7112a 2023-03-26 13:56:33,255 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/592b3b19d7eb4e1090097d80a0c03c77 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/592b3b19d7eb4e1090097d80a0c03c77 2023-03-26 13:56:33,256 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/13537a9fdd044af0b15ceaab62013c4f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/13537a9fdd044af0b15ceaab62013c4f 2023-03-26 13:56:33,257 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/b987176d581d4be092f8958fee226836 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/b987176d581d4be092f8958fee226836 2023-03-26 13:56:33,258 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/ce45cc607feb447580e0f9415641f53f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/ce45cc607feb447580e0f9415641f53f 2023-03-26 13:56:33,258 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/684a473d246b4fbb8b3f9d0996d32e43 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/684a473d246b4fbb8b3f9d0996d32e43 2023-03-26 13:56:33,259 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/68a0cefd47c949cd992ef3611c2cbf90 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/68a0cefd47c949cd992ef3611c2cbf90 2023-03-26 13:56:33,260 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/795b3597ae0e4b0ba5e6285e1cebb90b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/795b3597ae0e4b0ba5e6285e1cebb90b 2023-03-26 13:56:33,261 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/15cc4feea2304544953a476e27e1f90a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/15cc4feea2304544953a476e27e1f90a 2023-03-26 13:56:33,261 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/f5c9a26cddd24fbf9950540e374697ed to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/f5c9a26cddd24fbf9950540e374697ed 2023-03-26 13:56:33,262 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/aaef2502f97444d7bd46de64dca2be80 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/aaef2502f97444d7bd46de64dca2be80 2023-03-26 13:56:33,263 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/1308f8b1e31a473abdeb636f10e5c77c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/1308f8b1e31a473abdeb636f10e5c77c 2023-03-26 13:56:33,264 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/c8e812fc39c54d83816d095dd99c86ab to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/c8e812fc39c54d83816d095dd99c86ab 2023-03-26 13:56:33,265 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/a7fb8a70234a4cd08f7c84179546b1ea to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/a7fb8a70234a4cd08f7c84179546b1ea 2023-03-26 13:56:33,266 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/246e7b75270c45878fda2544e269e9c5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/246e7b75270c45878fda2544e269e9c5 2023-03-26 13:56:33,267 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5cc807e528bf47559b3da507a9a8b5b1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5cc807e528bf47559b3da507a9a8b5b1 2023-03-26 13:56:33,268 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/fdba608112264d99b20a35c81e4e801b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/fdba608112264d99b20a35c81e4e801b 2023-03-26 13:56:33,269 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/315cb44a48a247b8a05e0241650c373b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/315cb44a48a247b8a05e0241650c373b 2023-03-26 13:56:33,269 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/49a708e6928d43bf8a2f1a8f58abee24 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/49a708e6928d43bf8a2f1a8f58abee24 2023-03-26 13:56:33,270 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/35c7910ffb694d0c954c8716a7da2912 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/35c7910ffb694d0c954c8716a7da2912 2023-03-26 13:56:33,271 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/e2476138db9b4111916aea48abf77b23 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/e2476138db9b4111916aea48abf77b23 2023-03-26 13:56:33,272 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/04ce652266cc44248c9f965814de4abf to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/04ce652266cc44248c9f965814de4abf 2023-03-26 13:56:33,273 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/96cac0dbb3e3407fae9395dd56f62ceb to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/96cac0dbb3e3407fae9395dd56f62ceb 2023-03-26 13:56:33,273 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/4377046344354ee493e82e1d98f7d644 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/4377046344354ee493e82e1d98f7d644 2023-03-26 13:56:33,274 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/d9071412486c408eb5fc777fadfa8d11 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/d9071412486c408eb5fc777fadfa8d11 2023-03-26 13:56:33,275 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/4d303961e5eb43afa951f75f2f00d00a to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/4d303961e5eb43afa951f75f2f00d00a 2023-03-26 13:56:33,276 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/37f40c8fa6734189818d5a94ec97ae01 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/37f40c8fa6734189818d5a94ec97ae01 2023-03-26 13:56:33,277 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/7e590381548e4542ae928fcfbb9cc88d to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/7e590381548e4542ae928fcfbb9cc88d 2023-03-26 13:56:33,277 DEBUG [StoreCloser-TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5bddb7c33e20468485b757f7269fdfd4 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/5bddb7c33e20468485b757f7269fdfd4 2023-03-26 13:56:33,281 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/recovered.edits/1489.seqid, newMaxSeqId=1489, maxSeqId=4 2023-03-26 13:56:33,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048. 2023-03-26 13:56:33,282 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for f904470a09c7c6a52bdab844746de048: 2023-03-26 13:56:33,283 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:33,284 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=f904470a09c7c6a52bdab844746de048, regionState=CLOSED 2023-03-26 13:56:33,284 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679838993284"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679838993284"}]},"ts":"1679838993284"} 2023-03-26 13:56:33,285 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-03-26 13:56:33,286 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure f904470a09c7c6a52bdab844746de048, server=jenkins-hbase4.apache.org,42517,1679838814721 in 1.5310 sec 2023-03-26 13:56:33,286 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-03-26 13:56:33,287 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f904470a09c7c6a52bdab844746de048, UNASSIGN in 1.5340 sec 2023-03-26 13:56:33,287 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679838993287"}]},"ts":"1679838993287"} 2023-03-26 13:56:33,288 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-26 13:56:33,289 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-26 13:56:33,290 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5440 sec 2023-03-26 13:56:33,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-26 13:56:33,853 INFO [Listener at localhost/41531] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-03-26 13:56:33,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-26 13:56:33,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:56:33,856 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:56:33,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-26 13:56:33,857 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:56:33,860 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:33,862 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A, FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B, FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C, FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/recovered.edits] 2023-03-26 13:56:33,864 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/6a5929d7379842c4a3642d75a1e1835c to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/6a5929d7379842c4a3642d75a1e1835c 2023-03-26 13:56:33,865 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/780a30fea3a7459b9a6828bd795c6100 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/780a30fea3a7459b9a6828bd795c6100 2023-03-26 13:56:33,866 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/9a587fa13aaa4318a4f5f5537706c58e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/A/9a587fa13aaa4318a4f5f5537706c58e 2023-03-26 13:56:33,868 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/00534be0a532448b9aaa9a8e27f6eb54 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/00534be0a532448b9aaa9a8e27f6eb54 2023-03-26 13:56:33,870 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/5c86b0542d1a460b87f38e08eafecee3 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/5c86b0542d1a460b87f38e08eafecee3 2023-03-26 13:56:33,871 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/7207c61e929e47028d4aab61359ff494 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/B/7207c61e929e47028d4aab61359ff494 2023-03-26 13:56:33,873 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/17ef05bbf1294ba680c25d412fa594ff to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/17ef05bbf1294ba680c25d412fa594ff 2023-03-26 13:56:33,874 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/697f292327b8472faba000f047227e7b to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/697f292327b8472faba000f047227e7b 2023-03-26 13:56:33,875 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/d3e3ec2eef7d4bbea1e4847faf8421ba to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/C/d3e3ec2eef7d4bbea1e4847faf8421ba 2023-03-26 13:56:33,877 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/recovered.edits/1489.seqid to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048/recovered.edits/1489.seqid 2023-03-26 13:56:33,877 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/.tmp/data/default/TestAcidGuarantees/f904470a09c7c6a52bdab844746de048 2023-03-26 13:56:33,877 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 13:56:33,878 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-26 13:56:33,878 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-26 13:56:33,881 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260167229dbb524db69b6dcaeead4ca5f3 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260167229dbb524db69b6dcaeead4ca5f3 2023-03-26 13:56:33,881 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260cfdebd83f714483b89c448664676823 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260cfdebd83f714483b89c448664676823 2023-03-26 13:56:33,882 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260ed61c8676cb4a21aa1d9fa47f9111cc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260ed61c8676cb4a21aa1d9fa47f9111cc 2023-03-26 13:56:33,883 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032612ddbf404d6447fda82f6cae81ed94b1 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032612ddbf404d6447fda82f6cae81ed94b1 2023-03-26 13:56:33,884 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303261a801abe30a34f718ea82384f577a592 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303261a801abe30a34f718ea82384f577a592 2023-03-26 13:56:33,885 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303261b47c3c9890640869038377bc47d12ff to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303261b47c3c9890640869038377bc47d12ff 2023-03-26 13:56:33,886 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303261b8de2364e1249639eed2ea356f9a01e to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303261b8de2364e1249639eed2ea356f9a01e 2023-03-26 13:56:33,886 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032638101cf7bd514654977e23dc5076c3ee to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032638101cf7bd514654977e23dc5076c3ee 2023-03-26 13:56:33,887 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263d92ca8bc92b40abaa200a2a6b9e5b96 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263d92ca8bc92b40abaa200a2a6b9e5b96 2023-03-26 13:56:33,888 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032641f8e2dc534446c9a533501d75dbe709 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032641f8e2dc534446c9a533501d75dbe709 2023-03-26 13:56:33,889 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326555a8924acb546b4972ff48d7c53c7f5 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326555a8924acb546b4972ff48d7c53c7f5 2023-03-26 13:56:33,890 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303265ce1a662100b42db9029d9e3994b4d38 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303265ce1a662100b42db9029d9e3994b4d38 2023-03-26 13:56:33,891 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326630c1a7002b84477840c9e4b9ee53f18 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326630c1a7002b84477840c9e4b9ee53f18 2023-03-26 13:56:33,892 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032679246fe630bc4c0ab96051a2c9ab2ffd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032679246fe630bc4c0ab96051a2c9ab2ffd 2023-03-26 13:56:33,893 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303268f73597ee445476684a760a19f96aec8 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303268f73597ee445476684a760a19f96aec8 2023-03-26 13:56:33,894 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303269127725d10ec4457be7c2fe3bd79ce3f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303269127725d10ec4457be7c2fe3bd79ce3f 2023-03-26 13:56:33,894 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303269255ecfef8364bd8be2cd1cb331b2ecd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303269255ecfef8364bd8be2cd1cb331b2ecd 2023-03-26 13:56:33,895 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032698c457567f2c4d75b01b4b4e67231520 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032698c457567f2c4d75b01b4b4e67231520 2023-03-26 13:56:33,896 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303269d17e50fde8d40ff9e23741b54f35dbd to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303269d17e50fde8d40ff9e23741b54f35dbd 2023-03-26 13:56:33,897 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a47259ff82ea44aab43e772fed276603 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a47259ff82ea44aab43e772fed276603 2023-03-26 13:56:33,898 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a6565602e610409a910620c92ee321b2 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a6565602e610409a910620c92ee321b2 2023-03-26 13:56:33,899 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a6b0d40ee5df460eaaa365d76dde1703 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a6b0d40ee5df460eaaa365d76dde1703 2023-03-26 13:56:33,899 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ab23e4033d264f3f91eaba2b077f22fc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ab23e4033d264f3f91eaba2b077f22fc 2023-03-26 13:56:33,900 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326b7e1a5203ec24c76895bb59588956dba to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326b7e1a5203ec24c76895bb59588956dba 2023-03-26 13:56:33,901 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ba3b171520064a0eb74175b394dfb9ae to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ba3b171520064a0eb74175b394dfb9ae 2023-03-26 13:56:33,902 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326d0157d9a2df24945aed2161c860337ae to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326d0157d9a2df24945aed2161c860337ae 2023-03-26 13:56:33,902 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326d54f2bbdca384a39bfd1112650f32745 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326d54f2bbdca384a39bfd1112650f32745 2023-03-26 13:56:33,903 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e6fc74309e8841e296a48cf29830699f to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e6fc74309e8841e296a48cf29830699f 2023-03-26 13:56:33,904 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e7fbefe4d9ab4fe087df153f78bb3424 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e7fbefe4d9ab4fe087df153f78bb3424 2023-03-26 13:56:33,905 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ed9201018fe54bba98fd6c2743a18718 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ed9201018fe54bba98fd6c2743a18718 2023-03-26 13:56:33,906 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ef58c0d97cae4a8d813e44ec9c093424 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ef58c0d97cae4a8d813e44ec9c093424 2023-03-26 13:56:33,906 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f1e7838f89974be88251a91f6660c9fc to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f1e7838f89974be88251a91f6660c9fc 2023-03-26 13:56:33,907 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f737d4a1ec834650979c48940809d140 to hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f737d4a1ec834650979c48940809d140 2023-03-26 13:56:33,908 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-26 13:56:33,909 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:56:33,911 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-26 13:56:33,912 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-26 13:56:33,913 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:56:33,913 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-26 13:56:33,913 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679838993913"}]},"ts":"9223372036854775807"} 2023-03-26 13:56:33,914 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-26 13:56:33,914 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => f904470a09c7c6a52bdab844746de048, NAME => 'TestAcidGuarantees,,1679838966004.f904470a09c7c6a52bdab844746de048.', STARTKEY => '', ENDKEY => ''}] 2023-03-26 13:56:33,914 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-26 13:56:33,914 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679838993914"}]},"ts":"9223372036854775807"} 2023-03-26 13:56:33,915 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-26 13:56:33,918 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 13:56:33,919 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 64 msec 2023-03-26 13:56:33,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45485] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-26 13:56:33,958 INFO [Listener at localhost/41531] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-03-26 13:56:33,969 INFO [Listener at localhost/41531] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 235), OpenFileDescriptor=606 (was 608), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=436 (was 412) - SystemLoadAverage LEAK? -, ProcessCount=171 (was 171), AvailableMemoryMB=9143 (was 8996) - AvailableMemoryMB LEAK? - 2023-03-26 13:56:33,969 INFO [Listener at localhost/41531] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-26 13:56:33,970 INFO [Listener at localhost/41531] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-26 13:56:33,970 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x164139d4 to 127.0.0.1:54332 2023-03-26 13:56:33,970 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:33,971 DEBUG [Listener at localhost/41531] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-26 13:56:33,971 DEBUG [Listener at localhost/41531] util.JVMClusterUtil(257): Found active master hash=549836153, stopped=false 2023-03-26 13:56:33,971 INFO [Listener at localhost/41531] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,45485,1679838813504 2023-03-26 13:56:33,973 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-26 13:56:33,973 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-26 13:56:33,973 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 13:56:33,973 INFO [Listener at localhost/41531] procedure2.ProcedureExecutor(629): Stopping 2023-03-26 13:56:33,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-26 13:56:33,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-26 13:56:33,974 DEBUG [Listener at localhost/41531] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x46da7ed8 to 127.0.0.1:54332 2023-03-26 13:56:33,974 DEBUG [Listener at localhost/41531] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:33,974 INFO [Listener at localhost/41531] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,42517,1679838814721' ***** 2023-03-26 13:56:33,974 INFO [Listener at localhost/41531] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-26 13:56:33,974 INFO [RS:0;jenkins-hbase4:42517] regionserver.HeapMemoryManager(220): Stopping 2023-03-26 13:56:33,974 INFO [RS:0;jenkins-hbase4:42517] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-26 13:56:33,975 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-26 13:56:33,975 INFO [RS:0;jenkins-hbase4:42517] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-26 13:56:33,975 INFO [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(3304): Received CLOSE for 98e3213d37305f33190db3afbc13024b 2023-03-26 13:56:33,975 INFO [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:33,975 DEBUG [RS:0;jenkins-hbase4:42517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0cbf7cd6 to 127.0.0.1:54332 2023-03-26 13:56:33,975 DEBUG [RS:0;jenkins-hbase4:42517] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:33,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 98e3213d37305f33190db3afbc13024b, disabling compactions & flushes 2023-03-26 13:56:33,976 INFO [RS:0;jenkins-hbase4:42517] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-26 13:56:33,976 INFO [RS:0;jenkins-hbase4:42517] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-26 13:56:33,976 INFO [RS:0;jenkins-hbase4:42517] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-26 13:56:33,976 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b. 2023-03-26 13:56:33,976 INFO [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-26 13:56:33,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b. 2023-03-26 13:56:33,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b. after waiting 0 ms 2023-03-26 13:56:33,976 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b. 2023-03-26 13:56:33,976 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 98e3213d37305f33190db3afbc13024b 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-26 13:56:33,976 INFO [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-03-26 13:56:33,976 DEBUG [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 98e3213d37305f33190db3afbc13024b=hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b.} 2023-03-26 13:56:33,977 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-26 13:56:33,977 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-26 13:56:33,977 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-26 13:56:33,977 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-26 13:56:33,977 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-26 13:56:33,977 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-03-26 13:56:33,979 DEBUG [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1505): Waiting on 1588230740, 98e3213d37305f33190db3afbc13024b 2023-03-26 13:56:34,038 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-26 13:56:34,061 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-03-26 13:56:34,061 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-03-26 13:56:34,179 DEBUG [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1505): Waiting on 1588230740, 98e3213d37305f33190db3afbc13024b 2023-03-26 13:56:34,379 DEBUG [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1505): Waiting on 1588230740, 98e3213d37305f33190db3afbc13024b 2023-03-26 13:56:34,384 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/namespace/98e3213d37305f33190db3afbc13024b/.tmp/info/f2a8774886b544679b008e325e90bfc4 2023-03-26 13:56:34,384 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/.tmp/info/e633f31d25c74b3d85cd509aa74ad074 2023-03-26 13:56:34,388 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/namespace/98e3213d37305f33190db3afbc13024b/.tmp/info/f2a8774886b544679b008e325e90bfc4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/namespace/98e3213d37305f33190db3afbc13024b/info/f2a8774886b544679b008e325e90bfc4 2023-03-26 13:56:34,388 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e633f31d25c74b3d85cd509aa74ad074 2023-03-26 13:56:34,391 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/namespace/98e3213d37305f33190db3afbc13024b/info/f2a8774886b544679b008e325e90bfc4, entries=2, sequenceid=6, filesize=4.8 K 2023-03-26 13:56:34,392 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 98e3213d37305f33190db3afbc13024b in 416ms, sequenceid=6, compaction requested=false 2023-03-26 13:56:34,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/namespace/98e3213d37305f33190db3afbc13024b/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-26 13:56:34,399 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b. 2023-03-26 13:56:34,399 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 98e3213d37305f33190db3afbc13024b: 2023-03-26 13:56:34,399 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1679838816626.98e3213d37305f33190db3afbc13024b. 2023-03-26 13:56:34,579 DEBUG [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-26 13:56:34,780 DEBUG [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-26 13:56:34,797 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/.tmp/rep_barrier/34c41334c1d54e18a9ce1d57fb173a92 2023-03-26 13:56:34,800 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 34c41334c1d54e18a9ce1d57fb173a92 2023-03-26 13:56:34,980 INFO [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-03-26 13:56:34,980 DEBUG [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-03-26 13:56:34,980 DEBUG [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-26 13:56:35,180 DEBUG [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-26 13:56:35,205 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/.tmp/table/726142813b8540c6bc361cc9242b3c94 2023-03-26 13:56:35,208 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 726142813b8540c6bc361cc9242b3c94 2023-03-26 13:56:35,209 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/.tmp/info/e633f31d25c74b3d85cd509aa74ad074 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/info/e633f31d25c74b3d85cd509aa74ad074 2023-03-26 13:56:35,213 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for e633f31d25c74b3d85cd509aa74ad074 2023-03-26 13:56:35,213 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/info/e633f31d25c74b3d85cd509aa74ad074, entries=22, sequenceid=93, filesize=7.1 K 2023-03-26 13:56:35,214 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/.tmp/rep_barrier/34c41334c1d54e18a9ce1d57fb173a92 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/rep_barrier/34c41334c1d54e18a9ce1d57fb173a92 2023-03-26 13:56:35,217 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 34c41334c1d54e18a9ce1d57fb173a92 2023-03-26 13:56:35,217 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/rep_barrier/34c41334c1d54e18a9ce1d57fb173a92, entries=6, sequenceid=93, filesize=5.5 K 2023-03-26 13:56:35,218 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/.tmp/table/726142813b8540c6bc361cc9242b3c94 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/table/726142813b8540c6bc361cc9242b3c94 2023-03-26 13:56:35,220 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 726142813b8540c6bc361cc9242b3c94 2023-03-26 13:56:35,221 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/table/726142813b8540c6bc361cc9242b3c94, entries=9, sequenceid=93, filesize=5.5 K 2023-03-26 13:56:35,221 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 1244ms, sequenceid=93, compaction requested=false 2023-03-26 13:56:35,230 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-03-26 13:56:35,231 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-26 13:56:35,232 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-26 13:56:35,232 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-26 13:56:35,232 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-26 13:56:35,381 INFO [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,42517,1679838814721; all regions closed. 2023-03-26 13:56:35,389 DEBUG [RS:0;jenkins-hbase4:42517] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/oldWALs 2023-03-26 13:56:35,389 INFO [RS:0;jenkins-hbase4:42517] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42517%2C1679838814721.meta:.meta(num 1679838816375) 2023-03-26 13:56:35,393 DEBUG [RS:0;jenkins-hbase4:42517] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/oldWALs 2023-03-26 13:56:35,393 INFO [RS:0;jenkins-hbase4:42517] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C42517%2C1679838814721:(num 1679838816189) 2023-03-26 13:56:35,393 DEBUG [RS:0;jenkins-hbase4:42517] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:35,393 INFO [RS:0;jenkins-hbase4:42517] regionserver.LeaseManager(133): Closed leases 2023-03-26 13:56:35,394 INFO [RS:0;jenkins-hbase4:42517] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-03-26 13:56:35,394 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-26 13:56:35,395 INFO [RS:0;jenkins-hbase4:42517] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:42517 2023-03-26 13:56:35,400 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,42517,1679838814721 2023-03-26 13:56:35,400 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-26 13:56:35,401 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-26 13:56:35,401 ERROR [Listener at localhost/41531-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3aa90903 rejected from java.util.concurrent.ThreadPoolExecutor@3e968a8e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2816] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-26 13:56:35,402 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,42517,1679838814721] 2023-03-26 13:56:35,403 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,42517,1679838814721; numProcessing=1 2023-03-26 13:56:35,405 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,42517,1679838814721 already deleted, retry=false 2023-03-26 13:56:35,405 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,42517,1679838814721 expired; onlineServers=0 2023-03-26 13:56:35,405 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,45485,1679838813504' ***** 2023-03-26 13:56:35,405 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-26 13:56:35,406 DEBUG [M:0;jenkins-hbase4:45485] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@61f8df42, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-26 13:56:35,406 INFO [M:0;jenkins-hbase4:45485] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,45485,1679838813504 2023-03-26 13:56:35,406 INFO [M:0;jenkins-hbase4:45485] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,45485,1679838813504; all regions closed. 2023-03-26 13:56:35,406 DEBUG [M:0;jenkins-hbase4:45485] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 13:56:35,406 DEBUG [M:0;jenkins-hbase4:45485] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-26 13:56:35,406 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-26 13:56:35,406 DEBUG [M:0;jenkins-hbase4:45485] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-26 13:56:35,406 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679838815901] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679838815901,5,FailOnTimeoutGroup] 2023-03-26 13:56:35,406 INFO [M:0;jenkins-hbase4:45485] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-26 13:56:35,406 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679838815900] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679838815900,5,FailOnTimeoutGroup] 2023-03-26 13:56:35,406 INFO [M:0;jenkins-hbase4:45485] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-26 13:56:35,406 INFO [M:0;jenkins-hbase4:45485] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-03-26 13:56:35,407 DEBUG [M:0;jenkins-hbase4:45485] master.HMaster(1512): Stopping service threads 2023-03-26 13:56:35,407 INFO [M:0;jenkins-hbase4:45485] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-26 13:56:35,407 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-03-26 13:56:35,407 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 13:56:35,407 ERROR [M:0;jenkins-hbase4:45485] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-03-26 13:56:35,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-26 13:56:35,408 INFO [M:0;jenkins-hbase4:45485] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-26 13:56:35,408 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-26 13:56:35,408 DEBUG [M:0;jenkins-hbase4:45485] zookeeper.ZKUtil(398): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-26 13:56:35,408 WARN [M:0;jenkins-hbase4:45485] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-26 13:56:35,408 INFO [M:0;jenkins-hbase4:45485] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-26 13:56:35,408 INFO [M:0;jenkins-hbase4:45485] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-26 13:56:35,409 DEBUG [M:0;jenkins-hbase4:45485] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-26 13:56:35,409 INFO [M:0;jenkins-hbase4:45485] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-26 13:56:35,409 DEBUG [M:0;jenkins-hbase4:45485] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-26 13:56:35,409 DEBUG [M:0;jenkins-hbase4:45485] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-26 13:56:35,409 DEBUG [M:0;jenkins-hbase4:45485] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-26 13:56:35,409 INFO [M:0;jenkins-hbase4:45485] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.65 KB heapSize=393.05 KB 2023-03-26 13:56:35,425 INFO [M:0;jenkins-hbase4:45485] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.65 KB at sequenceid=573 (bloomFilter=true), to=hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/5df14c96260e4f74a2237d937eb589b4 2023-03-26 13:56:35,429 INFO [M:0;jenkins-hbase4:45485] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5df14c96260e4f74a2237d937eb589b4 2023-03-26 13:56:35,430 DEBUG [M:0;jenkins-hbase4:45485] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/5df14c96260e4f74a2237d937eb589b4 as hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/5df14c96260e4f74a2237d937eb589b4 2023-03-26 13:56:35,434 INFO [M:0;jenkins-hbase4:45485] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5df14c96260e4f74a2237d937eb589b4 2023-03-26 13:56:35,434 INFO [M:0;jenkins-hbase4:45485] regionserver.HStore(1080): Added hdfs://localhost:40005/user/jenkins/test-data/cd9a6f83-1b6d-dcce-1148-9b2fa59a6366/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/5df14c96260e4f74a2237d937eb589b4, entries=65, sequenceid=573, filesize=24.7 K 2023-03-26 13:56:35,434 INFO [M:0;jenkins-hbase4:45485] regionserver.HRegion(2948): Finished flush of dataSize ~344.65 KB/352924, heapSize ~393.04 KB/402472, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 25ms, sequenceid=573, compaction requested=false 2023-03-26 13:56:35,435 INFO [M:0;jenkins-hbase4:45485] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-26 13:56:35,436 DEBUG [M:0;jenkins-hbase4:45485] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-26 13:56:35,438 INFO [M:0;jenkins-hbase4:45485] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-26 13:56:35,438 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-26 13:56:35,439 INFO [M:0;jenkins-hbase4:45485] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:45485 2023-03-26 13:56:35,440 DEBUG [M:0;jenkins-hbase4:45485] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,45485,1679838813504 already deleted, retry=false 2023-03-26 13:56:35,504 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-26 13:56:35,504 INFO [RS:0;jenkins-hbase4:42517] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,42517,1679838814721; zookeeper connection closed. 2023-03-26 13:56:35,504 ERROR [Listener at localhost/41531-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@38367ca5 rejected from java.util.concurrent.ThreadPoolExecutor@3e968a8e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2816] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-26 13:56:35,505 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): regionserver:42517-0x101211a03bf0001, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-26 13:56:35,505 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@42e1cdf6] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@42e1cdf6 2023-03-26 13:56:35,505 ERROR [Listener at localhost/41531-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@40edba91 rejected from java.util.concurrent.ThreadPoolExecutor@3e968a8e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2816] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-26 13:56:35,506 INFO [Listener at localhost/41531] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-03-26 13:56:35,604 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-26 13:56:35,604 INFO [M:0;jenkins-hbase4:45485] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,45485,1679838813504; zookeeper connection closed. 2023-03-26 13:56:35,604 ERROR [Listener at localhost/41531-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@41fd6ecc rejected from java.util.concurrent.ThreadPoolExecutor@5d130203[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4698] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-26 13:56:35,605 DEBUG [Listener at localhost/41531-EventThread] zookeeper.ZKWatcher(600): master:45485-0x101211a03bf0000, quorum=127.0.0.1:54332, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-26 13:56:35,605 ERROR [Listener at localhost/41531-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2970a6e5 rejected from java.util.concurrent.ThreadPoolExecutor@5d130203[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4698] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-26 13:56:35,606 WARN [Listener at localhost/41531] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-26 13:56:35,610 INFO [Listener at localhost/41531] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-26 13:56:35,718 WARN [BP-1173788531-172.31.14.131-1679838810147 heartbeating to localhost/127.0.0.1:40005] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-26 13:56:35,718 WARN [BP-1173788531-172.31.14.131-1679838810147 heartbeating to localhost/127.0.0.1:40005] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1173788531-172.31.14.131-1679838810147 (Datanode Uuid 8c3178b5-de1c-466e-9d5b-1432272d7f2d) service to localhost/127.0.0.1:40005 2023-03-26 13:56:35,722 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/cluster_18e80c40-7651-1a2f-6095-ce10559d351d/dfs/data/data1/current/BP-1173788531-172.31.14.131-1679838810147] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-26 13:56:35,722 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/51aba9ae-3970-56b2-cb2d-28841f9d5b18/cluster_18e80c40-7651-1a2f-6095-ce10559d351d/dfs/data/data2/current/BP-1173788531-172.31.14.131-1679838810147] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-26 13:56:35,759 INFO [Listener at localhost/41531] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-26 13:56:35,934 INFO [Listener at localhost/41531] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-26 13:56:36,126 INFO [Listener at localhost/41531] hbase.HBaseTestingUtility(1293): Minicluster is down